[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260620 22:59:41.351222 10778 test_util.cc:268] Using random seed: 1565338856
I20260620 22:59:41.361792 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 22:59:41.361867 10778 ts_itest-base.cc:113] --------------
I20260620 22:59:41.361934 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 22:59:41.362020 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 22:59:41.362087 10778 ts_itest-base.cc:116] --------------
2026-06-20T22:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T22:59:41Z Disabled control of system clock
I20260620 22:59:41.383154 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:37361
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:37361
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260620 22:59:41.688081 10790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:41.688510 10790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:41.688690 10790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:41.695962 10790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 22:59:41.696053 10790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:41.696094 10790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 22:59:41.696147 10790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 22:59:41.705175 10790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:37361
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 22:59:41.706441 10790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:41.708163 10790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:41.719779 10795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:41.719820 10798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:41.719856 10796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:41.720577 10790 server_base.cc:1034] running on GCE node
I20260620 22:59:41.721299 10790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:41.722621 10790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:41.723834 10790 hybrid_clock.cc:648] HybridClock initialized: now 1781996381723757 us; error 51 us; skew 500 ppm
I20260620 22:59:41.724177 10790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 22:59:41.729537 10790 webserver.cc:460] Webserver started at http://127.10.134.190:39827/ using document root <none> and password file <none>
I20260620 22:59:41.730278 10790 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:41.730408 10790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:41.730710 10790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:41.733299 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "354504e2f1694e74a7bea41989d18405"
format_stamp: "Formatted at 2026-06-20 22:59:41 on dist-test-slave-8k25"
I20260620 22:59:41.734083 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "354504e2f1694e74a7bea41989d18405"
format_stamp: "Formatted at 2026-06-20 22:59:41 on dist-test-slave-8k25"
I20260620 22:59:41.738940 10790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 22:59:41.742213 10804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:41.743479 10790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:41.743647 10790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "354504e2f1694e74a7bea41989d18405"
format_stamp: "Formatted at 2026-06-20 22:59:41 on dist-test-slave-8k25"
I20260620 22:59:41.743834 10790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:41.762684 10790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:41.763509 10790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:41.763823 10790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:41.781422 10790 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:37361
I20260620 22:59:41.781451 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:37361 every 8 connection(s)
I20260620 22:59:41.782781 10790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20260620 22:59:41.786522 10856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:41.786819 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 10790
I20260620 22:59:41.787045 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20260620 22:59:41.796542 10856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405: Bootstrap starting.
I20260620 22:59:41.799415 10856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:41.800384 10856 log.cc:824] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:41.803016 10856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405: No bootstrap required, opened a new log
I20260620 22:59:41.811661 10856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354504e2f1694e74a7bea41989d18405" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 37361 } }
I20260620 22:59:41.812031 10856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 22:59:41.812157 10856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354504e2f1694e74a7bea41989d18405, State: Initialized, Role: FOLLOWER
I20260620 22:59:41.812881 10856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354504e2f1694e74a7bea41989d18405" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 37361 } }
I20260620 22:59:41.813251 10856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 22:59:41.813400 10856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 22:59:41.813613 10856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:41.815884 10856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354504e2f1694e74a7bea41989d18405" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 37361 } }
I20260620 22:59:41.816411 10856 leader_election.cc:302] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 354504e2f1694e74a7bea41989d18405; no voters:
I20260620 22:59:41.817062 10856 leader_election.cc:288] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 22:59:41.817198 10861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:41.817891 10861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [term 1 LEADER]: Becoming Leader. State: Replica: 354504e2f1694e74a7bea41989d18405, State: Running, Role: LEADER
I20260620 22:59:41.818427 10861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354504e2f1694e74a7bea41989d18405" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 37361 } }
I20260620 22:59:41.818970 10856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 22:59:41.821295 10862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "354504e2f1694e74a7bea41989d18405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354504e2f1694e74a7bea41989d18405" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 37361 } } }
I20260620 22:59:41.821800 10862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:41.821705 10863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 354504e2f1694e74a7bea41989d18405. Latest consensus state: current_term: 1 leader_uuid: "354504e2f1694e74a7bea41989d18405" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354504e2f1694e74a7bea41989d18405" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 37361 } } }
I20260620 22:59:41.822048 10863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405 [sys.catalog]: This master's current role is: LEADER
I20260620 22:59:41.823643 10866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 22:59:41.828991 10866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 22:59:41.837599 10866 catalog_manager.cc:1347] Generated new cluster ID: f96d0bed43734b049c9274888955d9ea
I20260620 22:59:41.837715 10866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 22:59:41.866806 10866 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 22:59:41.868070 10866 catalog_manager.cc:1504] Loading token signing keys...
I20260620 22:59:41.874989 10866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 354504e2f1694e74a7bea41989d18405: Generated new TSK 0
I20260620 22:59:41.875773 10866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 22:59:41.882939 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--builtin_ntp_servers=127.10.134.148:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260620 22:59:42.204532 10880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:59:42.204808 10880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:42.204865 10880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:42.204926 10880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:42.212002 10880 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260620 22:59:42.212088 10880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:42.212196 10880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 22:59:42.220697 10880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 22:59:42.221798 10880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:42.223409 10880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:42.235514 10886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:42.235514 10885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:42.236004 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:42.236280 10880 server_base.cc:1034] running on GCE node
I20260620 22:59:42.236843 10880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:42.237677 10880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:42.238903 10880 hybrid_clock.cc:648] HybridClock initialized: now 1781996382238809 us; error 76 us; skew 500 ppm
I20260620 22:59:42.239194 10880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 22:59:42.241237 10880 webserver.cc:460] Webserver started at http://127.10.134.129:45365/ using document root <none> and password file <none>
I20260620 22:59:42.241879 10880 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:42.242043 10880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:42.242321 10880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:42.244709 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd"
format_stamp: "Formatted at 2026-06-20 22:59:42 on dist-test-slave-8k25"
I20260620 22:59:42.245395 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd"
format_stamp: "Formatted at 2026-06-20 22:59:42 on dist-test-slave-8k25"
I20260620 22:59:42.250291 10880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 22:59:42.253310 10894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:42.254598 10880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 22:59:42.254760 10880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd"
format_stamp: "Formatted at 2026-06-20 22:59:42 on dist-test-slave-8k25"
I20260620 22:59:42.254940 10880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:42.273816 10880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:42.274590 10880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:42.274856 10880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:42.275676 10880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:42.276945 10880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:42.277045 10880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:42.277189 10880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:42.277259 10880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:42.308097 10880 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:38405
I20260620 22:59:42.308137 11006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:38405 every 8 connection(s)
I20260620 22:59:42.309365 10880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20260620 22:59:42.319402 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 10880
I20260620 22:59:42.319614 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20260620 22:59:42.322775 11007 heartbeater.cc:347] Connected to a master server at 127.10.134.190:37361
I20260620 22:59:42.323086 11007 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:42.323102 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--builtin_ntp_servers=127.10.134.148:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260620 22:59:42.323896 11007 heartbeater.cc:511] Master 127.10.134.190:37361 requested a full tablet report, sending...
I20260620 22:59:42.325894 10821 ts_manager.cc:194] Registered new tserver with Master: 4a0ff4ad41c64b14bd0f1deefe2630fd (127.10.134.129:38405)
I20260620 22:59:42.328032 10821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:44511
W20260620 22:59:42.630270 11010 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:59:42.630540 11010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:42.630595 11010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:42.630662 11010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:42.637416 11010 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260620 22:59:42.637504 11010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:42.637611 11010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 22:59:42.646064 11010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 22:59:42.647383 11010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:42.649122 11010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:42.661291 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:42.661312 11018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:42.661247 11016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:42.661898 11010 server_base.cc:1034] running on GCE node
I20260620 22:59:42.662822 11010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:42.663661 11010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:42.664896 11010 hybrid_clock.cc:648] HybridClock initialized: now 1781996382664835 us; error 35 us; skew 500 ppm
I20260620 22:59:42.665174 11010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 22:59:42.667256 11010 webserver.cc:460] Webserver started at http://127.10.134.130:33875/ using document root <none> and password file <none>
I20260620 22:59:42.667922 11010 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:42.668059 11010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:42.668330 11010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:42.670802 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "f92ed924edb14662afe3e5927f1fe89b"
format_stamp: "Formatted at 2026-06-20 22:59:42 on dist-test-slave-8k25"
I20260620 22:59:42.671516 11010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "f92ed924edb14662afe3e5927f1fe89b"
format_stamp: "Formatted at 2026-06-20 22:59:42 on dist-test-slave-8k25"
I20260620 22:59:42.676213 11010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 22:59:42.679301 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:42.680437 11010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 22:59:42.680603 11010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "f92ed924edb14662afe3e5927f1fe89b"
format_stamp: "Formatted at 2026-06-20 22:59:42 on dist-test-slave-8k25"
I20260620 22:59:42.680805 11010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:42.696203 11010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:42.696923 11010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:42.697193 11010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:42.698087 11010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:42.699352 11010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:42.699447 11010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:42.699587 11010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:42.699656 11010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:42.731998 11010 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:43595
I20260620 22:59:42.732020 11136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:43595 every 8 connection(s)
I20260620 22:59:42.733240 11010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20260620 22:59:42.738193 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11010
I20260620 22:59:42.738356 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20260620 22:59:42.741153 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--builtin_ntp_servers=127.10.134.148:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260620 22:59:42.746763 11137 heartbeater.cc:347] Connected to a master server at 127.10.134.190:37361
I20260620 22:59:42.747079 11137 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:42.747857 11137 heartbeater.cc:511] Master 127.10.134.190:37361 requested a full tablet report, sending...
I20260620 22:59:42.749220 10821 ts_manager.cc:194] Registered new tserver with Master: f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595)
I20260620 22:59:42.750056 10821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:44327
W20260620 22:59:43.051798 11140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:59:43.052139 11140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:43.052248 11140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:43.052390 11140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:43.059441 11140 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260620 22:59:43.059567 11140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:43.059742 11140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 22:59:43.068609 11140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 22:59:43.069818 11140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:43.071525 11140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:43.083513 11146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:43.083607 11145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:43.083722 11148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:43.084730 11140 server_base.cc:1034] running on GCE node
I20260620 22:59:43.085268 11140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:43.086174 11140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:43.087376 11140 hybrid_clock.cc:648] HybridClock initialized: now 1781996383087323 us; error 31 us; skew 500 ppm
I20260620 22:59:43.087688 11140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 22:59:43.089764 11140 webserver.cc:460] Webserver started at http://127.10.134.131:38265/ using document root <none> and password file <none>
I20260620 22:59:43.090447 11140 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:43.090579 11140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:43.090878 11140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:43.093235 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0"
format_stamp: "Formatted at 2026-06-20 22:59:43 on dist-test-slave-8k25"
I20260620 22:59:43.093971 11140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0"
format_stamp: "Formatted at 2026-06-20 22:59:43 on dist-test-slave-8k25"
I20260620 22:59:43.098645 11140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 22:59:43.101678 11154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:43.102911 11140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 22:59:43.103080 11140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0"
format_stamp: "Formatted at 2026-06-20 22:59:43 on dist-test-slave-8k25"
I20260620 22:59:43.103271 11140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:43.127789 11140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:43.128624 11140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:43.128901 11140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:43.129805 11140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:43.131210 11140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:43.131311 11140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:43.131448 11140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:43.131518 11140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:43.164191 11140 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:34351
I20260620 22:59:43.164278 11266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:34351 every 8 connection(s)
I20260620 22:59:43.165529 11140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20260620 22:59:43.166481 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11140
I20260620 22:59:43.166659 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20260620 22:59:43.178581 11267 heartbeater.cc:347] Connected to a master server at 127.10.134.190:37361
I20260620 22:59:43.178853 11267 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:43.179469 11267 heartbeater.cc:511] Master 127.10.134.190:37361 requested a full tablet report, sending...
I20260620 22:59:43.180725 10821 ts_manager.cc:194] Registered new tserver with Master: bbebd1115f6c45c5b5c9cf1a9adf47d0 (127.10.134.131:34351)
I20260620 22:59:43.181603 10821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:50585
I20260620 22:59:43.183123 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 22:59:43.187145 10778 test_util.cc:268] Using random seed: 1567174820
I20260620 22:59:43.210013 10821 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35220:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 22:59:43.211530 10821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 22:59:43.241549 11202 tablet_service.cc:1467] Processing CreateTablet for tablet 8d050e69bf0c450bb30bce9f006813f1 (DEFAULT_TABLE table=test-workload [id=93b5a72ccb4f490e90b0fc60ed645f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:43.241835 10942 tablet_service.cc:1467] Processing CreateTablet for tablet 8d050e69bf0c450bb30bce9f006813f1 (DEFAULT_TABLE table=test-workload [id=93b5a72ccb4f490e90b0fc60ed645f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:43.241835 11072 tablet_service.cc:1467] Processing CreateTablet for tablet 8d050e69bf0c450bb30bce9f006813f1 (DEFAULT_TABLE table=test-workload [id=93b5a72ccb4f490e90b0fc60ed645f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 22:59:43.242808 11202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d050e69bf0c450bb30bce9f006813f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:43.243141 10942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d050e69bf0c450bb30bce9f006813f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:43.243141 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d050e69bf0c450bb30bce9f006813f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 22:59:43.253635 11285 tablet_bootstrap.cc:492] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: Bootstrap starting.
I20260620 22:59:43.256675 11286 tablet_bootstrap.cc:492] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: Bootstrap starting.
I20260620 22:59:43.256809 11285 tablet_bootstrap.cc:654] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:43.258061 11285 log.cc:824] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:43.258610 11287 tablet_bootstrap.cc:492] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: Bootstrap starting.
I20260620 22:59:43.260288 11286 tablet_bootstrap.cc:654] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:43.261368 11286 log.cc:824] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:43.261732 11287 tablet_bootstrap.cc:654] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: Neither blocks nor log segments found. Creating new log.
I20260620 22:59:43.262189 11285 tablet_bootstrap.cc:492] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: No bootstrap required, opened a new log
I20260620 22:59:43.262473 11285 ts_tablet_manager.cc:1397] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 22:59:43.262859 11287 log.cc:824] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: Log is configured to *not* fsync() on all Append() calls
I20260620 22:59:43.264183 11286 tablet_bootstrap.cc:492] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: No bootstrap required, opened a new log
I20260620 22:59:43.264415 11286 ts_tablet_manager.cc:1397] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 22:59:43.265146 11287 tablet_bootstrap.cc:492] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: No bootstrap required, opened a new log
I20260620 22:59:43.265398 11287 ts_tablet_manager.cc:1397] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 22:59:43.272192 11287 raft_consensus.cc:348] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.272542 11287 raft_consensus.cc:729] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0ff4ad41c64b14bd0f1deefe2630fd, State: Initialized, Role: FOLLOWER
I20260620 22:59:43.272933 11287 consensus_queue.cc:260] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.273155 11285 raft_consensus.cc:348] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.273552 11285 raft_consensus.cc:729] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbebd1115f6c45c5b5c9cf1a9adf47d0, State: Initialized, Role: FOLLOWER
I20260620 22:59:43.274088 11007 heartbeater.cc:503] Master 127.10.134.190:37361 was elected leader, sending a full tablet report...
I20260620 22:59:43.273963 11285 consensus_queue.cc:260] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.274724 11287 ts_tablet_manager.cc:1428] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260620 22:59:43.275039 11267 heartbeater.cc:503] Master 127.10.134.190:37361 was elected leader, sending a full tablet report...
I20260620 22:59:43.275388 11286 raft_consensus.cc:348] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.275748 11286 raft_consensus.cc:729] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92ed924edb14662afe3e5927f1fe89b, State: Initialized, Role: FOLLOWER
I20260620 22:59:43.275882 11285 ts_tablet_manager.cc:1428] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260620 22:59:43.276125 11286 consensus_queue.cc:260] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.277208 11137 heartbeater.cc:503] Master 127.10.134.190:37361 was elected leader, sending a full tablet report...
I20260620 22:59:43.277774 11286 ts_tablet_manager.cc:1428] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 22:59:43.311687 11008 tablet.cc:2369] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:59:43.324342 11132 generic_service.cc:139] {username='slave'} at 127.0.0.1:47072 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260620 22:59:43.324549 11132 generic_service.cc:175] {username='slave'} at 127.0.0.1:47072 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.10.134.130:43595'
W20260620 22:59:43.325918 11132 generic_service.cc:139] {username='slave'} at 127.0.0.1:47072 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:43.326104 11132 generic_service.cc:175] {username='slave'} at 127.0.0.1:47072 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260620 22:59:43.332297 11262 generic_service.cc:139] {username='slave'} at 127.0.0.1:59270 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260620 22:59:43.332506 11262 generic_service.cc:175] {username='slave'} at 127.0.0.1:59270 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.10.134.130:43595'
W20260620 22:59:43.333700 11262 generic_service.cc:139] {username='slave'} at 127.0.0.1:59270 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:43.333822 11262 generic_service.cc:175] {username='slave'} at 127.0.0.1:59270 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260620 22:59:43.339597 11002 generic_service.cc:139] {username='slave'} at 127.0.0.1:49176 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260620 22:59:43.339798 11002 generic_service.cc:175] {username='slave'} at 127.0.0.1:49176 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.10.134.130:43595'
W20260620 22:59:43.340987 11002 generic_service.cc:139] {username='slave'} at 127.0.0.1:49176 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:43.341130 11002 generic_service.cc:175] {username='slave'} at 127.0.0.1:49176 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260620 22:59:43.342404 11222 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d050e69bf0c450bb30bce9f006813f1"
dest_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0"
from {username='slave'} at 127.0.0.1:59258
I20260620 22:59:43.342715 11222 raft_consensus.cc:482] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 22:59:43.342921 11222 raft_consensus.cc:3037] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:43.344997 11222 raft_consensus.cc:504] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.347309 11222 leader_election.cc:288] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [CANDIDATE]: Term 1 election: Requested vote from peers 4a0ff4ad41c64b14bd0f1deefe2630fd (127.10.134.129:38405), f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595)
W20260620 22:59:43.347936 11296 leader_election.cc:334] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595
I20260620 22:59:43.352914 10962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d050e69bf0c450bb30bce9f006813f1" candidate_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd"
I20260620 22:59:43.353271 10962 raft_consensus.cc:3037] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:43.355765 10962 raft_consensus.cc:2454] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbebd1115f6c45c5b5c9cf1a9adf47d0 in term 1.
I20260620 22:59:43.356259 11156 leader_election.cc:302] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a0ff4ad41c64b14bd0f1deefe2630fd, bbebd1115f6c45c5b5c9cf1a9adf47d0; no voters: f92ed924edb14662afe3e5927f1fe89b
I20260620 22:59:43.356657 11292 raft_consensus.cc:2781] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 22:59:43.357178 11292 raft_consensus.cc:686] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 1 LEADER]: Becoming Leader. State: Replica: bbebd1115f6c45c5b5c9cf1a9adf47d0, State: Running, Role: LEADER
I20260620 22:59:43.357740 11292 consensus_queue.cc:237] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:43.362820 10820 catalog_manager.cc:5495] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbebd1115f6c45c5b5c9cf1a9adf47d0 (127.10.134.131). New cstate: current_term: 1 leader_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } health_report { overall_health: UNKNOWN } } }
W20260620 22:59:43.417372 11268 tablet.cc:2369] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 22:59:43.438851 11292 consensus_queue.cc:1035] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 22:59:43.444726 11296 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 1: this message will repeat every 5th retry.
W20260620 22:59:43.485152 11138 tablet.cc:2369] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 22:59:43.941155 11296 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 6: this message will repeat every 5th retry.
W20260620 22:59:44.386811 11296 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 11: this message will repeat every 5th retry.
W20260620 22:59:44.891786 11296 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 16: this message will repeat every 5th retry.
W20260620 22:59:45.453737 11296 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 21: this message will repeat every 5th retry.
W20260620 22:59:45.977550 11296 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 26: this message will repeat every 5th retry.
I20260620 22:59:46.374261 11297 consensus_queue.cc:579] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Leader has been unable to successfully communicate with peer f92ed924edb14662afe3e5927f1fe89b for more than 3 seconds (3.015s)
W20260620 22:59:46.377354 10821 catalog_manager.cc:5173] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d050e69bf0c450bb30bce9f006813f1 with cas_config_opid_index -1: no extra replica candidate found for tablet 8d050e69bf0c450bb30bce9f006813f1 (table test-workload [id=93b5a72ccb4f490e90b0fc60ed645f68]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20260620 22:59:46.434674 11132 generic_service.cc:139] {username='slave'} at 127.0.0.1:47072 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:46.434840 11132 generic_service.cc:175] {username='slave'} at 127.0.0.1:47072 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260620 22:59:46.435950 11262 generic_service.cc:139] {username='slave'} at 127.0.0.1:59270 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:46.436088 11262 generic_service.cc:175] {username='slave'} at 127.0.0.1:59270 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260620 22:59:46.437064 11002 generic_service.cc:139] {username='slave'} at 127.0.0.1:49176 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:46.437201 11002 generic_service.cc:175] {username='slave'} at 127.0.0.1:49176 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20260620 22:59:46.458284 11092 raft_consensus.cc:3037] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 0 FOLLOWER]: Advancing to term 1
I20260620 22:59:46.599572 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.132:0
--local_ip_for_outbound_sockets=127.10.134.132
--webserver_interface=127.10.134.132
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--builtin_ntp_servers=127.10.134.148:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260620 22:59:46.912252 11309 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 22:59:46.912557 11309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 22:59:46.912664 11309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 22:59:46.912807 11309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 22:59:46.919647 11309 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260620 22:59:46.919766 11309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 22:59:46.919909 11309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.132
I20260620 22:59:46.928627 11309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.134.132
--webserver_port=0
--tserver_master_addrs=127.10.134.190:37361
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.10.134.132
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 22:59:46.929818 11309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 22:59:46.931507 11309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 22:59:46.943697 11317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:46.943670 11315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 22:59:46.943807 11314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 22:59:46.944233 11309 server_base.cc:1034] running on GCE node
I20260620 22:59:46.944859 11309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 22:59:46.945770 11309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 22:59:46.947046 11309 hybrid_clock.cc:648] HybridClock initialized: now 1781996386946976 us; error 49 us; skew 500 ppm
I20260620 22:59:46.947359 11309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 22:59:46.952391 11309 webserver.cc:460] Webserver started at http://127.10.134.132:38265/ using document root <none> and password file <none>
I20260620 22:59:46.952998 11309 fs_manager.cc:359] Metadata directory not provided
I20260620 22:59:46.953109 11309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 22:59:46.953341 11309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 22:59:46.955773 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "eb5c81458de24d82bfafb66b209ef971"
format_stamp: "Formatted at 2026-06-20 22:59:46 on dist-test-slave-8k25"
I20260620 22:59:46.956481 11309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "eb5c81458de24d82bfafb66b209ef971"
format_stamp: "Formatted at 2026-06-20 22:59:46 on dist-test-slave-8k25"
I20260620 22:59:46.961843 11309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 22:59:46.965032 11323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:46.966333 11309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 22:59:46.966495 11309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "eb5c81458de24d82bfafb66b209ef971"
format_stamp: "Formatted at 2026-06-20 22:59:46 on dist-test-slave-8k25"
I20260620 22:59:46.966672 11309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 22:59:46.986164 11309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 22:59:46.986860 11309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 22:59:46.987187 11309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 22:59:46.988107 11309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 22:59:46.989399 11309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 22:59:46.989503 11309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:46.989631 11309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 22:59:46.989702 11309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 22:59:47.021852 11309 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.132:40821
I20260620 22:59:47.021884 11435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.132:40821 every 8 connection(s)
I20260620 22:59:47.023258 11309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20260620 22:59:47.023952 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11309
I20260620 22:59:47.024134 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20260620 22:59:47.036782 11436 heartbeater.cc:347] Connected to a master server at 127.10.134.190:37361
I20260620 22:59:47.037070 11436 heartbeater.cc:464] Registering TS with master...
I20260620 22:59:47.037681 11436 heartbeater.cc:511] Master 127.10.134.190:37361 requested a full tablet report, sending...
I20260620 22:59:47.039047 10821 ts_manager.cc:194] Registered new tserver with Master: eb5c81458de24d82bfafb66b209ef971 (127.10.134.132:40821)
I20260620 22:59:47.039826 10821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.132:52423
I20260620 22:59:47.043618 11222 consensus_queue.cc:237] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } peers { permanent_uuid: "eb5c81458de24d82bfafb66b209ef971" member_type: NON_VOTER last_known_addr { host: "127.10.134.132" port: 40821 } attrs { promote: true } }
I20260620 22:59:47.046260 11092 raft_consensus.cc:1261] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 1 FOLLOWER]: Refusing update from remote peer bbebd1115f6c45c5b5c9cf1a9adf47d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:47.046892 11302 consensus_queue.cc:1035] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 22:59:47.047125 10962 raft_consensus.cc:1261] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 1 FOLLOWER]: Refusing update from remote peer bbebd1115f6c45c5b5c9cf1a9adf47d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 22:59:47.047665 11302 consensus_queue.cc:1035] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 22:59:47.049736 11302 raft_consensus.cc:2932] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } peers { permanent_uuid: "eb5c81458de24d82bfafb66b209ef971" member_type: NON_VOTER last_known_addr { host: "127.10.134.132" port: 40821 } attrs { promote: true } } }
I20260620 22:59:47.050199 11092 raft_consensus.cc:2932] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } peers { permanent_uuid: "eb5c81458de24d82bfafb66b209ef971" member_type: NON_VOTER last_known_addr { host: "127.10.134.132" port: 40821 } attrs { promote: true } } }
I20260620 22:59:47.051740 10962 raft_consensus.cc:2932] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } peers { permanent_uuid: "eb5c81458de24d82bfafb66b209ef971" member_type: NON_VOTER last_known_addr { host: "127.10.134.132" port: 40821 } attrs { promote: true } } }
I20260620 22:59:47.054090 10807 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d050e69bf0c450bb30bce9f006813f1 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20260620 22:59:47.055676 10821 catalog_manager.cc:5495] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 reported cstate change: config changed from index -1 to 2, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) added. New cstate: current_term: 1 leader_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5c81458de24d82bfafb66b209ef971" member_type: NON_VOTER last_known_addr { host: "127.10.134.132" port: 40821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260620 22:59:47.058094 11156 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer eb5c81458de24d82bfafb66b209ef971 (127.10.134.132:40821): Couldn't send request to peer eb5c81458de24d82bfafb66b209ef971. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d050e69bf0c450bb30bce9f006813f1. This is attempt 1: this message will repeat every 5th retry.
I20260620 22:59:47.064366 11222 consensus_queue.cc:237] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } }
I20260620 22:59:47.066043 11092 raft_consensus.cc:1261] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 1 FOLLOWER]: Refusing update from remote peer bbebd1115f6c45c5b5c9cf1a9adf47d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 22:59:47.066128 10962 raft_consensus.cc:1261] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 1 FOLLOWER]: Refusing update from remote peer bbebd1115f6c45c5b5c9cf1a9adf47d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 22:59:47.066507 11297 consensus_queue.cc:1035] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 22:59:47.066970 11297 consensus_queue.cc:1035] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 22:59:47.068553 11302 raft_consensus.cc:2932] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } }
I20260620 22:59:47.069113 11092 raft_consensus.cc:2932] T 8d050e69bf0c450bb30bce9f006813f1 P f92ed924edb14662afe3e5927f1fe89b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } }
I20260620 22:59:47.069391 10962 raft_consensus.cc:2932] T 8d050e69bf0c450bb30bce9f006813f1 P 4a0ff4ad41c64b14bd0f1deefe2630fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } } }
I20260620 22:59:47.071969 10807 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8d050e69bf0c450bb30bce9f006813f1 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260620 22:59:47.072934 10818 catalog_manager.cc:5495] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 reported cstate change: config changed from index 2 to 3, NON_VOTER eb5c81458de24d82bfafb66b209ef971 (127.10.134.132) evicted. New cstate: current_term: 1 leader_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a0ff4ad41c64b14bd0f1deefe2630fd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f92ed924edb14662afe3e5927f1fe89b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43595 } health_report { overall_health: HEALTHY } } }
W20260620 22:59:47.079243 11132 generic_service.cc:139] {username='slave'} at 127.0.0.1:47072 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:47.079385 11132 generic_service.cc:175] {username='slave'} at 127.0.0.1:47072 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260620 22:59:47.080683 11262 generic_service.cc:139] {username='slave'} at 127.0.0.1:59270 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:47.080847 11262 generic_service.cc:175] {username='slave'} at 127.0.0.1:59270 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260620 22:59:47.081990 11002 generic_service.cc:139] {username='slave'} at 127.0.0.1:49176 forcing change of non-runtime-safe flag fail_dns_resolution
I20260620 22:59:47.082106 11002 generic_service.cc:175] {username='slave'} at 127.0.0.1:49176 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260620 22:59:47.083300 11222 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d050e69bf0c450bb30bce9f006813f1"
dest_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0"
mode: GRACEFUL
new_leader_uuid: "bbebd1115f6c45c5b5c9cf1a9adf47d0"
from {username='slave'} at 127.0.0.1:59258
I20260620 22:59:47.083590 11222 raft_consensus.cc:595] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 [term 1 LEADER]: Received request to transfer leadership to TS bbebd1115f6c45c5b5c9cf1a9adf47d0
I20260620 22:59:47.083812 11371 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d050e69bf0c450bb30bce9f006813f1 with delete_type TABLET_DATA_TOMBSTONED (TS eb5c81458de24d82bfafb66b209ef971 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38144
W20260620 22:59:47.086915 10806 catalog_manager.cc:4808] TS eb5c81458de24d82bfafb66b209ef971 (127.10.134.132:40821): delete failed for tablet 8d050e69bf0c450bb30bce9f006813f1 because the tablet was not found. No further retry: Not found: Tablet not found: 8d050e69bf0c450bb30bce9f006813f1
I20260620 22:59:48.042153 11436 heartbeater.cc:503] Master 127.10.134.190:37361 was elected leader, sending a full tablet report...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_election-itest.cc:291: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
I20260620 23:00:17.171782 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 10880
I20260620 23:00:17.173024 11002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:17.263242 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 10880
I20260620 23:00:17.281728 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11010
I20260620 23:00:17.282667 11132 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 23:00:17.330402 11156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.129:38405: connect: Connection refused (error 111)
W20260620 23:00:17.331736 11156 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer 4a0ff4ad41c64b14bd0f1deefe2630fd (127.10.134.129:38405): Couldn't send request to peer 4a0ff4ad41c64b14bd0f1deefe2630fd. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:38405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:00:17.372421 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11010
W20260620 23:00:17.395143 11158 connection.cc:535] client connection to 127.10.134.130:43595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:00:17.395627 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11140
W20260620 23:00:17.395800 11462 consensus_peers.cc:487] T 8d050e69bf0c450bb30bce9f006813f1 P bbebd1115f6c45c5b5c9cf1a9adf47d0 -> Peer f92ed924edb14662afe3e5927f1fe89b (127.10.134.130:43595): Couldn't send request to peer f92ed924edb14662afe3e5927f1fe89b. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.10.134.130:43595. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:00:17.396600 11262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:17.487373 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11140
I20260620 23:00:17.505081 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11309
I20260620 23:00:17.512054 11431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:17.594338 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11309
I20260620 23:00:17.610325 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 10790
I20260620 23:00:17.611411 10851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:17.718179 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 10790
2026-06-20T23:00:17Z chronyd exiting
I20260620 23:00:17.751871 10778 test_util.cc:175] -----------------------------------------------
I20260620 23:00:17.751987 10778 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1781996381279935-10778-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (36406 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20260620 23:00:17.754349 10778 test_util.cc:268] Using random seed: 1601742025
I20260620 23:00:17.756060 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:00:17.756165 10778 ts_itest-base.cc:113] --------------
I20260620 23:00:17.756242 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:00:17.756309 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:00:17.756356 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:17Z Disabled control of system clock
I20260620 23:00:17.777127 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:35465
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:35465 with env {}
W20260620 23:00:18.083760 11477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:18.084017 11477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:18.084089 11477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:18.091018 11477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:18.091110 11477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:18.091150 11477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:18.091202 11477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:18.099939 11477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:35465
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:18.101107 11477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:18.102747 11477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:18.114739 11485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:18.115000 11477 server_base.cc:1034] running on GCE node
W20260620 23:00:18.115229 11482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:18.115275 11483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:18.116194 11477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:18.117503 11477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:18.118717 11477 hybrid_clock.cc:648] HybridClock initialized: now 1781996418118647 us; error 46 us; skew 500 ppm
I20260620 23:00:18.119057 11477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:18.121165 11477 webserver.cc:460] Webserver started at http://127.10.134.190:44677/ using document root <none> and password file <none>
I20260620 23:00:18.121789 11477 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:18.121963 11477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:18.122287 11477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:18.124763 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6"
format_stamp: "Formatted at 2026-06-20 23:00:18 on dist-test-slave-8k25"
I20260620 23:00:18.125490 11477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6"
format_stamp: "Formatted at 2026-06-20 23:00:18 on dist-test-slave-8k25"
I20260620 23:00:18.130326 11477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:00:18.133855 11491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:18.135178 11477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:00:18.135411 11477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6"
format_stamp: "Formatted at 2026-06-20 23:00:18 on dist-test-slave-8k25"
I20260620 23:00:18.135607 11477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:18.147621 11477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:18.148379 11477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:18.148679 11477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:18.166031 11477 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:35465
I20260620 23:00:18.166064 11542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:35465 every 8 connection(s)
I20260620 23:00:18.167260 11477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:00:18.170539 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11477
I20260620 23:00:18.170764 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:00:18.171336 11543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:18.183768 11543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6: Bootstrap starting.
I20260620 23:00:18.186579 11543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:18.187497 11543 log.cc:824] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:18.189869 11543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6: No bootstrap required, opened a new log
I20260620 23:00:18.196633 11543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35465 } }
I20260620 23:00:18.196964 11543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:18.197072 11543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fdf48f0a3b4c079e4c8eec0dd546b6, State: Initialized, Role: FOLLOWER
I20260620 23:00:18.197621 11543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35465 } }
I20260620 23:00:18.197835 11543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:18.197988 11543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:18.198160 11543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:18.199980 11543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35465 } }
I20260620 23:00:18.200419 11543 leader_election.cc:302] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30fdf48f0a3b4c079e4c8eec0dd546b6; no voters:
I20260620 23:00:18.201102 11543 leader_election.cc:288] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:18.201187 11548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:18.201788 11548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [term 1 LEADER]: Becoming Leader. State: Replica: 30fdf48f0a3b4c079e4c8eec0dd546b6, State: Running, Role: LEADER
I20260620 23:00:18.202378 11548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35465 } }
I20260620 23:00:18.203174 11543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:18.205243 11549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35465 } } }
I20260620 23:00:18.205246 11550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30fdf48f0a3b4c079e4c8eec0dd546b6. Latest consensus state: current_term: 1 leader_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fdf48f0a3b4c079e4c8eec0dd546b6" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35465 } } }
I20260620 23:00:18.205703 11549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:18.205698 11550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:18.212152 11554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:18.217320 11554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:18.224525 11554 catalog_manager.cc:1347] Generated new cluster ID: a365754b17c44d12bccb215e87c0f31d
I20260620 23:00:18.224628 11554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:18.246652 11554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:18.247846 11554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:18.254108 11554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30fdf48f0a3b4c079e4c8eec0dd546b6: Generated new TSK 0
I20260620 23:00:18.254976 11554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:18.268025 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35465
--builtin_ntp_servers=127.10.134.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260620 23:00:18.582127 11567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:18.582384 11567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:18.582535 11567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:18.589416 11567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:18.589610 11567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:00:18.598439 11567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35465
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:18.599625 11567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:18.601265 11567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:18.613878 11575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:18.613996 11572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:18.613965 11573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:18.614849 11567 server_base.cc:1034] running on GCE node
I20260620 23:00:18.615401 11567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:18.616199 11567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:18.617453 11567 hybrid_clock.cc:648] HybridClock initialized: now 1781996418617388 us; error 47 us; skew 500 ppm
I20260620 23:00:18.617751 11567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:18.622997 11567 webserver.cc:460] Webserver started at http://127.10.134.129:41425/ using document root <none> and password file <none>
I20260620 23:00:18.623625 11567 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:18.623719 11567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:18.623945 11567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:18.626433 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "096462170f9543fb82a18dcda9b41bbe"
format_stamp: "Formatted at 2026-06-20 23:00:18 on dist-test-slave-8k25"
I20260620 23:00:18.627100 11567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "096462170f9543fb82a18dcda9b41bbe"
format_stamp: "Formatted at 2026-06-20 23:00:18 on dist-test-slave-8k25"
I20260620 23:00:18.631718 11567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:00:18.634851 11581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:18.636015 11567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:00:18.636200 11567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "096462170f9543fb82a18dcda9b41bbe"
format_stamp: "Formatted at 2026-06-20 23:00:18 on dist-test-slave-8k25"
I20260620 23:00:18.636384 11567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:18.661746 11567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:18.662490 11567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:18.662791 11567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:18.663672 11567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:18.665026 11567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:18.665136 11567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:18.665274 11567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:18.665350 11567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:18.698137 11567 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:36119
I20260620 23:00:18.698176 11693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:36119 every 8 connection(s)
I20260620 23:00:18.699364 11567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:00:18.704638 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11567
I20260620 23:00:18.704821 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:00:18.708159 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35465
--builtin_ntp_servers=127.10.134.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260620 23:00:18.713773 11694 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35465
I20260620 23:00:18.714093 11694 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:18.714814 11694 heartbeater.cc:511] Master 127.10.134.190:35465 requested a full tablet report, sending...
I20260620 23:00:18.716660 11508 ts_manager.cc:194] Registered new tserver with Master: 096462170f9543fb82a18dcda9b41bbe (127.10.134.129:36119)
I20260620 23:00:18.718146 11508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:60479
W20260620 23:00:19.014797 11698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:19.015056 11698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:19.015187 11698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:19.022073 11698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:19.022275 11698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:00:19.038182 11698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35465
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:19.039734 11698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:19.041837 11698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:19.055083 11706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:19.055083 11704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:19.055521 11703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:19.055961 11698 server_base.cc:1034] running on GCE node
I20260620 23:00:19.056576 11698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:19.057443 11698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:19.058647 11698 hybrid_clock.cc:648] HybridClock initialized: now 1781996419058591 us; error 28 us; skew 500 ppm
I20260620 23:00:19.058934 11698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:19.060950 11698 webserver.cc:460] Webserver started at http://127.10.134.130:42487/ using document root <none> and password file <none>
I20260620 23:00:19.061645 11698 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:19.061740 11698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:19.064689 11698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:19.068676 11698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bf4d322831394f6386d99cefd103c659"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:19.069450 11698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bf4d322831394f6386d99cefd103c659"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:19.074275 11698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 23:00:19.077369 11712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:19.078622 11698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:00:19.078789 11698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bf4d322831394f6386d99cefd103c659"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:19.078974 11698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:19.098546 11698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:19.099248 11698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:19.099541 11698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:19.100381 11698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:19.101683 11698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:19.101783 11698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:19.101919 11698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:19.102028 11698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 23:00:19.133946 11698 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:43675
I20260620 23:00:19.134037 11824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:43675 every 8 connection(s)
I20260620 23:00:19.135238 11698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:00:19.143795 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11698
I20260620 23:00:19.144032 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:00:19.147223 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35465
--builtin_ntp_servers=127.10.134.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260620 23:00:19.149232 11825 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35465
I20260620 23:00:19.149538 11825 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:19.150404 11825 heartbeater.cc:511] Master 127.10.134.190:35465 requested a full tablet report, sending...
I20260620 23:00:19.151887 11508 ts_manager.cc:194] Registered new tserver with Master: bf4d322831394f6386d99cefd103c659 (127.10.134.130:43675)
I20260620 23:00:19.152936 11508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:36291
W20260620 23:00:19.463104 11828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:19.463353 11828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:19.463434 11828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:19.470428 11828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:19.470578 11828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:00:19.479292 11828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35465
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:19.480494 11828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:19.482131 11828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:19.494692 11833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:19.494663 11834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:19.494612 11836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:19.495697 11828 server_base.cc:1034] running on GCE node
I20260620 23:00:19.496340 11828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:19.497185 11828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:19.498421 11828 hybrid_clock.cc:648] HybridClock initialized: now 1781996419498354 us; error 45 us; skew 500 ppm
I20260620 23:00:19.498705 11828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:19.500847 11828 webserver.cc:460] Webserver started at http://127.10.134.131:35223/ using document root <none> and password file <none>
I20260620 23:00:19.501495 11828 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:19.501632 11828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:19.501914 11828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:19.504613 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "19043760b73a4d5295fd61d65e7eed27"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:19.505342 11828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "19043760b73a4d5295fd61d65e7eed27"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:19.510231 11828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:00:19.513576 11842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:19.514765 11828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:00:19.514938 11828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "19043760b73a4d5295fd61d65e7eed27"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:19.515115 11828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:19.536055 11828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:19.536818 11828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:19.537122 11828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:19.538079 11828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:19.539525 11828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:19.539619 11828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:19.539784 11828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:19.539853 11828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:19.571561 11828 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:41917
I20260620 23:00:19.571615 11954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:41917 every 8 connection(s)
I20260620 23:00:19.572815 11828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:00:19.574223 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 11828
I20260620 23:00:19.574375 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:00:19.586563 11955 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35465
I20260620 23:00:19.586858 11955 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:19.587533 11955 heartbeater.cc:511] Master 127.10.134.190:35465 requested a full tablet report, sending...
I20260620 23:00:19.589017 11507 ts_manager.cc:194] Registered new tserver with Master: 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917)
I20260620 23:00:19.590097 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:19.590163 11507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:44335
I20260620 23:00:19.605518 11507 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44002:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260620 23:00:19.612183 11507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:19.652508 11760 tablet_service.cc:1467] Processing CreateTablet for tablet 63bea5a61b264b4f91624a32c033efa4 (DEFAULT_TABLE table=TestTable [id=408b7ff7b8464527960eb18e14b183b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:19.652508 11629 tablet_service.cc:1467] Processing CreateTablet for tablet 63bea5a61b264b4f91624a32c033efa4 (DEFAULT_TABLE table=TestTable [id=408b7ff7b8464527960eb18e14b183b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:19.653898 11629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bea5a61b264b4f91624a32c033efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:19.653898 11760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bea5a61b264b4f91624a32c033efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:19.655987 11890 tablet_service.cc:1467] Processing CreateTablet for tablet 63bea5a61b264b4f91624a32c033efa4 (DEFAULT_TABLE table=TestTable [id=408b7ff7b8464527960eb18e14b183b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:19.657058 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63bea5a61b264b4f91624a32c033efa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:19.666949 11973 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Bootstrap starting.
I20260620 23:00:19.670523 11973 tablet_bootstrap.cc:654] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:19.670653 11974 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: Bootstrap starting.
I20260620 23:00:19.671381 11975 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: Bootstrap starting.
I20260620 23:00:19.671818 11973 log.cc:824] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:19.674386 11974 tablet_bootstrap.cc:654] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:19.674605 11975 tablet_bootstrap.cc:654] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:19.674746 11973 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: No bootstrap required, opened a new log
I20260620 23:00:19.675009 11973 ts_tablet_manager.cc:1397] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:00:19.675392 11974 log.cc:824] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:19.675714 11975 log.cc:824] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:19.677748 11974 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: No bootstrap required, opened a new log
I20260620 23:00:19.678056 11974 ts_tablet_manager.cc:1397] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:00:19.679948 11975 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: No bootstrap required, opened a new log
I20260620 23:00:19.680195 11975 ts_tablet_manager.cc:1397] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260620 23:00:19.684984 11974 raft_consensus.cc:348] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.685464 11974 raft_consensus.cc:374] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:19.685600 11974 raft_consensus.cc:729] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096462170f9543fb82a18dcda9b41bbe, State: Initialized, Role: FOLLOWER
I20260620 23:00:19.685730 11973 raft_consensus.cc:348] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.686164 11973 raft_consensus.cc:374] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:19.686368 11973 raft_consensus.cc:729] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19043760b73a4d5295fd61d65e7eed27, State: Initialized, Role: FOLLOWER
I20260620 23:00:19.686298 11974 consensus_queue.cc:260] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.687029 11973 consensus_queue.cc:260] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.690615 11694 heartbeater.cc:503] Master 127.10.134.190:35465 was elected leader, sending a full tablet report...
I20260620 23:00:19.691896 11975 raft_consensus.cc:348] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.692443 11974 ts_tablet_manager.cc:1428] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 23:00:19.692504 11975 raft_consensus.cc:374] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:19.692680 11975 raft_consensus.cc:729] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4d322831394f6386d99cefd103c659, State: Initialized, Role: FOLLOWER
I20260620 23:00:19.693383 11975 consensus_queue.cc:260] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.694571 11825 heartbeater.cc:503] Master 127.10.134.190:35465 was elected leader, sending a full tablet report...
I20260620 23:00:19.695020 11975 ts_tablet_manager.cc:1428] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260620 23:00:19.695011 11973 ts_tablet_manager.cc:1428] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
I20260620 23:00:19.695348 11955 heartbeater.cc:503] Master 127.10.134.190:35465 was elected leader, sending a full tablet report...
W20260620 23:00:19.701422 11695 tablet.cc:2369] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:00:19.824506 11956 tablet.cc:2369] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:00:19.887162 11826 tablet.cc:2369] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:00:19.888734 11980 raft_consensus.cc:482] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:19.889020 11980 raft_consensus.cc:504] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.890869 11980 leader_election.cc:288] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf4d322831394f6386d99cefd103c659 (127.10.134.130:43675), 096462170f9543fb82a18dcda9b41bbe (127.10.134.129:36119)
I20260620 23:00:19.899036 11649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bea5a61b264b4f91624a32c033efa4" candidate_uuid: "19043760b73a4d5295fd61d65e7eed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "096462170f9543fb82a18dcda9b41bbe" is_pre_election: true
I20260620 23:00:19.899233 11780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bea5a61b264b4f91624a32c033efa4" candidate_uuid: "19043760b73a4d5295fd61d65e7eed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4d322831394f6386d99cefd103c659" is_pre_election: true
I20260620 23:00:19.899571 11649 raft_consensus.cc:2454] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19043760b73a4d5295fd61d65e7eed27 in term 0.
I20260620 23:00:19.899814 11780 raft_consensus.cc:2454] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19043760b73a4d5295fd61d65e7eed27 in term 0.
I20260620 23:00:19.900219 11843 leader_election.cc:302] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 096462170f9543fb82a18dcda9b41bbe, 19043760b73a4d5295fd61d65e7eed27; no voters:
I20260620 23:00:19.900669 11980 raft_consensus.cc:2781] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:19.900820 11980 raft_consensus.cc:482] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:19.900939 11980 raft_consensus.cc:3037] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:19.903092 11980 raft_consensus.cc:504] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.903757 11980 leader_election.cc:288] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [CANDIDATE]: Term 1 election: Requested vote from peers bf4d322831394f6386d99cefd103c659 (127.10.134.130:43675), 096462170f9543fb82a18dcda9b41bbe (127.10.134.129:36119)
I20260620 23:00:19.904156 11780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bea5a61b264b4f91624a32c033efa4" candidate_uuid: "19043760b73a4d5295fd61d65e7eed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf4d322831394f6386d99cefd103c659"
I20260620 23:00:19.904289 11649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bea5a61b264b4f91624a32c033efa4" candidate_uuid: "19043760b73a4d5295fd61d65e7eed27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "096462170f9543fb82a18dcda9b41bbe"
I20260620 23:00:19.904353 11780 raft_consensus.cc:3037] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:19.904488 11649 raft_consensus.cc:3037] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:19.907393 11649 raft_consensus.cc:2454] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19043760b73a4d5295fd61d65e7eed27 in term 1.
I20260620 23:00:19.907393 11780 raft_consensus.cc:2454] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19043760b73a4d5295fd61d65e7eed27 in term 1.
I20260620 23:00:19.907799 11843 leader_election.cc:302] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 096462170f9543fb82a18dcda9b41bbe, 19043760b73a4d5295fd61d65e7eed27; no voters:
I20260620 23:00:19.908041 11980 raft_consensus.cc:2781] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:19.908491 11980 raft_consensus.cc:686] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 1 LEADER]: Becoming Leader. State: Replica: 19043760b73a4d5295fd61d65e7eed27, State: Running, Role: LEADER
I20260620 23:00:19.908949 11980 consensus_queue.cc:237] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:19.913295 11508 catalog_manager.cc:5495] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131). New cstate: current_term: 1 leader_uuid: "19043760b73a4d5295fd61d65e7eed27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:19.936064 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:19.938634 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 096462170f9543fb82a18dcda9b41bbe to finish bootstrapping
I20260620 23:00:19.949066 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bf4d322831394f6386d99cefd103c659 to finish bootstrapping
I20260620 23:00:19.956836 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 19043760b73a4d5295fd61d65e7eed27 to finish bootstrapping
I20260620 23:00:19.990998 11649 raft_consensus.cc:1261] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 1 FOLLOWER]: Refusing update from remote peer 19043760b73a4d5295fd61d65e7eed27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:19.991065 11780 raft_consensus.cc:1261] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 1 FOLLOWER]: Refusing update from remote peer 19043760b73a4d5295fd61d65e7eed27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:00:19.991890 11983 consensus_queue.cc:1035] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:19.992230 11980 consensus_queue.cc:1035] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:20.006155 11988 mvcc.cc:205] Tried to move back new op lower bound from 7299057336274780160 to 7299057335951585280. Current Snapshot: MvccSnapshot[applied={T|T < 7299057336274780160}]
I20260620 23:00:20.167107 11890 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260620 23:00:20.167128 11629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:20.167698 11760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
30fdf48f0a3b4c079e4c8eec0dd546b6 | 127.10.134.190:35465 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
096462170f9543fb82a18dcda9b41bbe | 127.10.134.129:36119 | HEALTHY | <none> | 0 | 0
19043760b73a4d5295fd61d65e7eed27 | 127.10.134.131:41917 | HEALTHY | <none> | 1 | 0
bf4d322831394f6386d99cefd103c659 | 127.10.134.130:43675 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:36119
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:43675
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:41917
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:36119
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:43675
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:41917
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260620 23:00:20.216965 10778 log_verifier.cc:126] Checking tablet 63bea5a61b264b4f91624a32c033efa4
I20260620 23:00:20.256182 10778 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 63bea5a61b264b4f91624a32c033efa4
I20260620 23:00:20.521373 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11828
I20260620 23:00:20.522284 11949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:20.620028 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11828
W20260620 23:00:20.633325 11715 connection.cc:535] server connection from 127.10.134.131:36151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:00:20.634473 11780 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63bea5a61b264b4f91624a32c033efa4"
dest_uuid: "bf4d322831394f6386d99cefd103c659"
from {username='slave'} at 127.0.0.1:53516
I20260620 23:00:20.634766 11780 raft_consensus.cc:482] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:00:20.634918 11780 raft_consensus.cc:3037] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:00:20.636897 11780 raft_consensus.cc:504] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:20.638386 11780 leader_election.cc:288] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [CANDIDATE]: Term 2 election: Requested vote from peers 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917), 096462170f9543fb82a18dcda9b41bbe (127.10.134.129:36119)
W20260620 23:00:20.641462 11714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.131:41917: connect: Connection refused (error 111)
W20260620 23:00:20.651752 11714 leader_election.cc:334] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917): Network error: Client connection negotiation failed: client connection to 127.10.134.131:41917: connect: Connection refused (error 111)
I20260620 23:00:20.651896 11649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63bea5a61b264b4f91624a32c033efa4" candidate_uuid: "bf4d322831394f6386d99cefd103c659" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "096462170f9543fb82a18dcda9b41bbe"
I20260620 23:00:20.652143 11649 raft_consensus.cc:3037] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:00:20.654335 11649 raft_consensus.cc:2454] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf4d322831394f6386d99cefd103c659 in term 2.
I20260620 23:00:20.654788 11713 leader_election.cc:302] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 096462170f9543fb82a18dcda9b41bbe, bf4d322831394f6386d99cefd103c659; no voters: 19043760b73a4d5295fd61d65e7eed27
I20260620 23:00:20.656483 12031 raft_consensus.cc:2781] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 2 FOLLOWER]: Leader election won for term 2
W20260620 23:00:20.657408 11959 meta_cache.cc:302] tablet 63bea5a61b264b4f91624a32c033efa4: replica 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917) has failed: Network error: Client connection negotiation failed: client connection to 127.10.134.131:41917: connect: Connection refused (error 111)
I20260620 23:00:20.657521 12031 raft_consensus.cc:686] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [term 2 LEADER]: Becoming Leader. State: Replica: bf4d322831394f6386d99cefd103c659, State: Running, Role: LEADER
I20260620 23:00:20.658324 12031 consensus_queue.cc:237] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:20.661643 11505 catalog_manager.cc:5495] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 reported cstate change: term changed from 1 to 2, leader changed from 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131) to bf4d322831394f6386d99cefd103c659 (127.10.134.130). New cstate: current_term: 2 leader_uuid: "bf4d322831394f6386d99cefd103c659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:20.675446 11649 raft_consensus.cc:1261] T 63bea5a61b264b4f91624a32c033efa4 P 096462170f9543fb82a18dcda9b41bbe [term 2 FOLLOWER]: Refusing update from remote peer bf4d322831394f6386d99cefd103c659: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260620 23:00:20.676177 12033 consensus_queue.cc:1035] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 23:00:20.676566 11714 consensus_peers.cc:487] T 63bea5a61b264b4f91624a32c033efa4 P bf4d322831394f6386d99cefd103c659 -> Peer 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917): Couldn't send request to peer 19043760b73a4d5295fd61d65e7eed27. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:41917: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:00:20.688853 12040 mvcc.cc:205] Tried to move back new op lower bound from 7299057339079954432 to 7299057339021135872. Current Snapshot: MvccSnapshot[applied={T|T < 7299057339079954432}]
I20260620 23:00:20.752859 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:41917
--local_ip_for_outbound_sockets=127.10.134.131
--tserver_master_addrs=127.10.134.190:35465
--webserver_port=35223
--webserver_interface=127.10.134.131
--builtin_ntp_servers=127.10.134.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260620 23:00:21.074791 12045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:21.075088 12045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:21.075261 12045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:21.082122 12045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:21.082363 12045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:00:21.091260 12045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:41917
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=35223
--tserver_master_addrs=127.10.134.190:35465
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:21.092617 12045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:21.094699 12045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:21.108780 12050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:21.108738 12051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:21.108824 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:21.109607 12045 server_base.cc:1034] running on GCE node
I20260620 23:00:21.110379 12045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:21.111382 12045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:21.112630 12045 hybrid_clock.cc:648] HybridClock initialized: now 1781996421112534 us; error 62 us; skew 500 ppm
I20260620 23:00:21.112948 12045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:21.115417 12045 webserver.cc:460] Webserver started at http://127.10.134.131:35223/ using document root <none> and password file <none>
I20260620 23:00:21.116137 12045 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:21.116282 12045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:21.121441 12045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:00:21.124326 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:21.125725 12045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 23:00:21.125895 12045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "19043760b73a4d5295fd61d65e7eed27"
format_stamp: "Formatted at 2026-06-20 23:00:19 on dist-test-slave-8k25"
I20260620 23:00:21.126935 12045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:21.159557 12045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:21.160486 12045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:21.160810 12045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:21.161826 12045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:21.163842 12066 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:00:21.170230 12045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:00:21.170375 12045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 23:00:21.170523 12045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:00:21.173686 12045 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:00:21.173795 12045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260620 23:00:21.174300 12066 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Bootstrap starting.
I20260620 23:00:21.203004 12066 log.cc:824] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:21.215682 12045 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:41917
I20260620 23:00:21.215688 12173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:41917 every 8 connection(s)
I20260620 23:00:21.217257 12045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:00:21.222718 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 12045
I20260620 23:00:21.237666 12174 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35465
I20260620 23:00:21.238144 12174 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:21.238926 12174 heartbeater.cc:511] Master 127.10.134.190:35465 requested a full tablet report, sending...
I20260620 23:00:21.241650 11505 ts_manager.cc:194] Re-registered known tserver with Master: 19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917)
I20260620 23:00:21.243207 11505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:50107
I20260620 23:00:21.249752 12066 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:00:21.250319 12066 tablet_bootstrap.cc:492] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Bootstrap complete.
I20260620 23:00:21.251469 12066 ts_tablet_manager.cc:1397] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Time spent bootstrapping tablet: real 0.077s user 0.066s sys 0.009s
I20260620 23:00:21.257061 12066 raft_consensus.cc:348] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:21.258002 12066 raft_consensus.cc:729] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19043760b73a4d5295fd61d65e7eed27, State: Initialized, Role: FOLLOWER
I20260620 23:00:21.258668 12066 consensus_queue.cc:260] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf4d322831394f6386d99cefd103c659" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 43675 } } peers { permanent_uuid: "19043760b73a4d5295fd61d65e7eed27" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41917 } } peers { permanent_uuid: "096462170f9543fb82a18dcda9b41bbe" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36119 } }
I20260620 23:00:21.259779 12174 heartbeater.cc:503] Master 127.10.134.190:35465 was elected leader, sending a full tablet report...
I20260620 23:00:21.260416 12066 ts_tablet_manager.cc:1428] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260620 23:00:21.316200 11629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:21.324841 12108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:21.333257 11760 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
30fdf48f0a3b4c079e4c8eec0dd546b6 | 127.10.134.190:35465 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
096462170f9543fb82a18dcda9b41bbe | 127.10.134.129:36119 | HEALTHY | <none> | 0 | 0
19043760b73a4d5295fd61d65e7eed27 | 127.10.134.131:41917 | HEALTHY | <none> | 0 | 0
bf4d322831394f6386d99cefd103c659 | 127.10.134.130:43675 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:36119
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:43675
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:41917
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:36119
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:43675
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:41917
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 63bea5a61b264b4f91624a32c033efa4 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
bf4d322831394f6386d99cefd103c659 (127.10.134.130:43675): RUNNING [LEADER]
19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917): RUNNING
096462170f9543fb82a18dcda9b41bbe (127.10.134.129:36119): RUNNING
All reported replicas are:
A = bf4d322831394f6386d99cefd103c659
B = 19043760b73a4d5295fd61d65e7eed27
C = 096462170f9543fb82a18dcda9b41bbe
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260620 23:00:21.374584 10778 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260620 23:00:21.610742 11629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:21.617496 12108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:21.619506 11760 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260620 23:00:21.634364 12128 raft_consensus.cc:3037] T 63bea5a61b264b4f91624a32c033efa4 P 19043760b73a4d5295fd61d65e7eed27 [term 1 FOLLOWER]: Advancing to term 2
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
30fdf48f0a3b4c079e4c8eec0dd546b6 | 127.10.134.190:35465 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
096462170f9543fb82a18dcda9b41bbe | 127.10.134.129:36119 | HEALTHY | <none> | 0 | 0
19043760b73a4d5295fd61d65e7eed27 | 127.10.134.131:41917 | HEALTHY | <none> | 0 | 0
bf4d322831394f6386d99cefd103c659 | 127.10.134.130:43675 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:36119
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:43675
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:41917
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:36119
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:43675
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:41917
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 63bea5a61b264b4f91624a32c033efa4 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
bf4d322831394f6386d99cefd103c659 (127.10.134.130:43675): RUNNING [LEADER]
19043760b73a4d5295fd61d65e7eed27 (127.10.134.131:41917): RUNNING
096462170f9543fb82a18dcda9b41bbe (127.10.134.129:36119): RUNNING
All reported replicas are:
A = bf4d322831394f6386d99cefd103c659
B = 19043760b73a4d5295fd61d65e7eed27
C = 096462170f9543fb82a18dcda9b41bbe
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260620 23:00:21.677337 10778 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260620 23:00:21.689656 12229 mvcc.cc:205] Tried to move back new op lower bound from 7299057339240243200 to 7299057339021135872. Current Snapshot: MvccSnapshot[applied={T|T < 7299057339183108096}]
I20260620 23:00:21.977937 12108 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:21.978451 11629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:21.978705 11760 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
30fdf48f0a3b4c079e4c8eec0dd546b6 | 127.10.134.190:35465 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
096462170f9543fb82a18dcda9b41bbe | 127.10.134.129:36119 | HEALTHY | <none> | 0 | 0
19043760b73a4d5295fd61d65e7eed27 | 127.10.134.131:41917 | HEALTHY | <none> | 0 | 0
bf4d322831394f6386d99cefd103c659 | 127.10.134.130:43675 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:36119
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:43675
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:41917
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:36119
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:43675
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:41917
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38911 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260620 23:00:22.020296 10778 log_verifier.cc:126] Checking tablet 63bea5a61b264b4f91624a32c033efa4
I20260620 23:00:22.063623 10778 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 63bea5a61b264b4f91624a32c033efa4
I20260620 23:00:22.268309 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11567
I20260620 23:00:22.269140 11689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:22.381999 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11567
I20260620 23:00:22.404115 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11698
I20260620 23:00:22.405102 11819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:22.500797 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11698
I20260620 23:00:22.518205 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 12045
I20260620 23:00:22.524443 12169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:22.618999 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 12045
I20260620 23:00:22.636168 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 11477
I20260620 23:00:22.637221 11538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:22.747233 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 11477
2026-06-20T23:00:22Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (5035 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20260620 23:00:22.789637 10778 test_util.cc:268] Using random seed: 1606777327
I20260620 23:00:22.791489 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:00:22.791548 10778 ts_itest-base.cc:113] --------------
I20260620 23:00:22.791612 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:00:22.791669 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:00:22.791754 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:00:22Z Disabled control of system clock
I20260620 23:00:22.815842 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:43173
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:43173 with env {}
W20260620 23:00:23.127727 12277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:23.128015 12277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:23.128090 12277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:23.135066 12277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:00:23.135175 12277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:23.135219 12277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:00:23.135274 12277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:00:23.144129 12277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:43173
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:23.145397 12277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:23.147050 12277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:23.158618 12285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:23.158658 12283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:23.158556 12282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:23.159839 12277 server_base.cc:1034] running on GCE node
I20260620 23:00:23.160565 12277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:23.161887 12277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:23.163148 12277 hybrid_clock.cc:648] HybridClock initialized: now 1781996423163077 us; error 50 us; skew 500 ppm
I20260620 23:00:23.163476 12277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:23.165740 12277 webserver.cc:460] Webserver started at http://127.10.134.190:43027/ using document root <none> and password file <none>
I20260620 23:00:23.166505 12277 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:23.166646 12277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:23.166982 12277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:23.169628 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0150d2ce16de43f7b479d9beebd9f137"
format_stamp: "Formatted at 2026-06-20 23:00:23 on dist-test-slave-8k25"
I20260620 23:00:23.170434 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0150d2ce16de43f7b479d9beebd9f137"
format_stamp: "Formatted at 2026-06-20 23:00:23 on dist-test-slave-8k25"
I20260620 23:00:23.175997 12277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:00:23.179443 12291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:23.180682 12277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:00:23.180851 12277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0150d2ce16de43f7b479d9beebd9f137"
format_stamp: "Formatted at 2026-06-20 23:00:23 on dist-test-slave-8k25"
I20260620 23:00:23.181061 12277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:23.207252 12277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:23.208209 12277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:23.208530 12277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:23.226228 12277 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:43173
I20260620 23:00:23.226269 12342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:43173 every 8 connection(s)
I20260620 23:00:23.227519 12277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:00:23.231125 12343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:23.232010 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 12277
I20260620 23:00:23.232188 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:00:23.243919 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137: Bootstrap starting.
I20260620 23:00:23.247097 12343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:23.248100 12343 log.cc:824] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:23.250418 12343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137: No bootstrap required, opened a new log
I20260620 23:00:23.257364 12343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0150d2ce16de43f7b479d9beebd9f137" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43173 } }
I20260620 23:00:23.257774 12343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:23.257880 12343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0150d2ce16de43f7b479d9beebd9f137, State: Initialized, Role: FOLLOWER
I20260620 23:00:23.258526 12343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0150d2ce16de43f7b479d9beebd9f137" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43173 } }
I20260620 23:00:23.258751 12343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:00:23.258886 12343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:00:23.259064 12343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:23.260984 12343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0150d2ce16de43f7b479d9beebd9f137" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43173 } }
I20260620 23:00:23.261448 12343 leader_election.cc:302] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0150d2ce16de43f7b479d9beebd9f137; no voters:
I20260620 23:00:23.262053 12343 leader_election.cc:288] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:00:23.262248 12348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:23.263393 12348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [term 1 LEADER]: Becoming Leader. State: Replica: 0150d2ce16de43f7b479d9beebd9f137, State: Running, Role: LEADER
I20260620 23:00:23.263945 12348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0150d2ce16de43f7b479d9beebd9f137" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43173 } }
I20260620 23:00:23.264216 12343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:00:23.266578 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0150d2ce16de43f7b479d9beebd9f137. Latest consensus state: current_term: 1 leader_uuid: "0150d2ce16de43f7b479d9beebd9f137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0150d2ce16de43f7b479d9beebd9f137" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43173 } } }
I20260620 23:00:23.266961 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:23.267575 12349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0150d2ce16de43f7b479d9beebd9f137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0150d2ce16de43f7b479d9beebd9f137" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43173 } } }
I20260620 23:00:23.267890 12349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137 [sys.catalog]: This master's current role is: LEADER
I20260620 23:00:23.270089 12357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:00:23.274938 12357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:00:23.283252 12357 catalog_manager.cc:1347] Generated new cluster ID: 71ae4ee6955748d1b06f1a5baf8d5615
I20260620 23:00:23.283386 12357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:00:23.294001 12357 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:00:23.295279 12357 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:00:23.303438 12357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0150d2ce16de43f7b479d9beebd9f137: Generated new TSK 0
I20260620 23:00:23.304534 12357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:00:23.316960 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43173
--builtin_ntp_servers=127.10.134.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260620 23:00:23.658767 12368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:23.659039 12368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:23.659202 12368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:23.666592 12368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:23.666836 12368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:00:23.675943 12368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:23.677239 12368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:23.679006 12368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:23.692291 12373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:23.692341 12376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:23.692397 12374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:23.693472 12368 server_base.cc:1034] running on GCE node
I20260620 23:00:23.694208 12368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:23.695106 12368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:23.696321 12368 hybrid_clock.cc:648] HybridClock initialized: now 1781996423696258 us; error 34 us; skew 500 ppm
I20260620 23:00:23.696617 12368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:23.702975 12368 webserver.cc:460] Webserver started at http://127.10.134.129:43981/ using document root <none> and password file <none>
I20260620 23:00:23.703673 12368 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:23.703801 12368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:23.704111 12368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:23.706914 12368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2b1834d241b74f0f8152bb84b9e628dd"
format_stamp: "Formatted at 2026-06-20 23:00:23 on dist-test-slave-8k25"
I20260620 23:00:23.707656 12368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2b1834d241b74f0f8152bb84b9e628dd"
format_stamp: "Formatted at 2026-06-20 23:00:23 on dist-test-slave-8k25"
I20260620 23:00:23.712677 12368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 23:00:23.715939 12382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:23.717379 12368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:00:23.717577 12368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2b1834d241b74f0f8152bb84b9e628dd"
format_stamp: "Formatted at 2026-06-20 23:00:23 on dist-test-slave-8k25"
I20260620 23:00:23.717777 12368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:23.731262 12368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:23.732075 12368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:23.732368 12368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:23.733309 12368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:23.734735 12368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:23.734835 12368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:23.734978 12368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:23.735050 12368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:23.768251 12368 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:33119
I20260620 23:00:23.768276 12494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:33119 every 8 connection(s)
I20260620 23:00:23.769631 12368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:00:23.777209 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 12368
I20260620 23:00:23.777410 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:00:23.780540 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43173
--builtin_ntp_servers=127.10.134.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260620 23:00:23.783502 12495 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43173
I20260620 23:00:23.783828 12495 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:23.784825 12495 heartbeater.cc:511] Master 127.10.134.190:43173 requested a full tablet report, sending...
I20260620 23:00:23.787204 12308 ts_manager.cc:194] Registered new tserver with Master: 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:23.789438 12308 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:38399
W20260620 23:00:24.096932 12499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:24.097209 12499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:24.097296 12499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:24.106751 12499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:24.107008 12499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:00:24.118333 12499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:24.119577 12499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:24.121297 12499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:24.133812 12504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:24.133888 12507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:24.133849 12505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:24.134953 12499 server_base.cc:1034] running on GCE node
I20260620 23:00:24.135483 12499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:24.136308 12499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:24.137557 12499 hybrid_clock.cc:648] HybridClock initialized: now 1781996424137453 us; error 53 us; skew 500 ppm
I20260620 23:00:24.137841 12499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:24.140071 12499 webserver.cc:460] Webserver started at http://127.10.134.130:36367/ using document root <none> and password file <none>
I20260620 23:00:24.140702 12499 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:24.140794 12499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:24.141028 12499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:24.143527 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c707c04fc03d49da82f97968100c19d0"
format_stamp: "Formatted at 2026-06-20 23:00:24 on dist-test-slave-8k25"
I20260620 23:00:24.144203 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c707c04fc03d49da82f97968100c19d0"
format_stamp: "Formatted at 2026-06-20 23:00:24 on dist-test-slave-8k25"
I20260620 23:00:24.149091 12499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:00:24.152201 12513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:24.153463 12499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:00:24.153630 12499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c707c04fc03d49da82f97968100c19d0"
format_stamp: "Formatted at 2026-06-20 23:00:24 on dist-test-slave-8k25"
I20260620 23:00:24.153826 12499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:24.172968 12499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:24.173818 12499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:24.174139 12499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:24.175100 12499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:24.176497 12499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:24.176604 12499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:24.176752 12499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:24.176824 12499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:24.209110 12499 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:39755
I20260620 23:00:24.209152 12625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:39755 every 8 connection(s)
I20260620 23:00:24.210506 12499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:00:24.219831 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 12499
I20260620 23:00:24.219995 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:00:24.223372 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43173
--builtin_ntp_servers=127.10.134.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260620 23:00:24.223798 12626 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43173
I20260620 23:00:24.224108 12626 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:24.225001 12626 heartbeater.cc:511] Master 127.10.134.190:43173 requested a full tablet report, sending...
I20260620 23:00:24.226554 12308 ts_manager.cc:194] Registered new tserver with Master: c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755)
I20260620 23:00:24.227632 12308 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:49103
W20260620 23:00:24.548710 12629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:00:24.548995 12629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:00:24.549158 12629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:00:24.556414 12629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:00:24.556625 12629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:00:24.565644 12629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:00:24.567008 12629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:00:24.568778 12629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:00:24.581372 12634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:24.581403 12637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:00:24.581496 12635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:00:24.582134 12629 server_base.cc:1034] running on GCE node
I20260620 23:00:24.582691 12629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:00:24.583590 12629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:00:24.584794 12629 hybrid_clock.cc:648] HybridClock initialized: now 1781996424584723 us; error 54 us; skew 500 ppm
I20260620 23:00:24.585130 12629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:00:24.587314 12629 webserver.cc:460] Webserver started at http://127.10.134.131:43247/ using document root <none> and password file <none>
I20260620 23:00:24.587967 12629 fs_manager.cc:359] Metadata directory not provided
I20260620 23:00:24.588112 12629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:00:24.588403 12629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:00:24.591166 12629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e49a8d52878a43aa83df884ff96e687e"
format_stamp: "Formatted at 2026-06-20 23:00:24 on dist-test-slave-8k25"
I20260620 23:00:24.591889 12629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e49a8d52878a43aa83df884ff96e687e"
format_stamp: "Formatted at 2026-06-20 23:00:24 on dist-test-slave-8k25"
I20260620 23:00:24.596655 12629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:00:24.599814 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:24.601061 12629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:00:24.601223 12629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e49a8d52878a43aa83df884ff96e687e"
format_stamp: "Formatted at 2026-06-20 23:00:24 on dist-test-slave-8k25"
I20260620 23:00:24.601404 12629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:00:24.627152 12629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:00:24.627948 12629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:00:24.628242 12629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:00:24.629136 12629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:00:24.630564 12629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:00:24.630667 12629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:24.630816 12629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:00:24.630887 12629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:00:24.664146 12629 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:44497
I20260620 23:00:24.664201 12755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:44497 every 8 connection(s)
I20260620 23:00:24.665962 12629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:00:24.666764 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 12629
I20260620 23:00:24.666941 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:00:24.680454 12756 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43173
I20260620 23:00:24.680711 12756 heartbeater.cc:464] Registering TS with master...
I20260620 23:00:24.681811 12756 heartbeater.cc:511] Master 127.10.134.190:43173 requested a full tablet report, sending...
I20260620 23:00:24.683295 12308 ts_manager.cc:194] Registered new tserver with Master: e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:24.684028 12308 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:38245
I20260620 23:00:24.694625 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:00:24.697631 10778 test_util.cc:268] Using random seed: 1608685306
I20260620 23:00:24.716569 12308 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33304:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:00:24.718169 12308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:00:24.745596 12430 tablet_service.cc:1467] Processing CreateTablet for tablet c4447a4bda0d41f09b88af81b2668b51 (DEFAULT_TABLE table=test-workload [id=5f35bf577cb14bce9aa74c0aa35a2da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:24.747198 12430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4447a4bda0d41f09b88af81b2668b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:24.748980 12561 tablet_service.cc:1467] Processing CreateTablet for tablet c4447a4bda0d41f09b88af81b2668b51 (DEFAULT_TABLE table=test-workload [id=5f35bf577cb14bce9aa74c0aa35a2da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:24.750603 12691 tablet_service.cc:1467] Processing CreateTablet for tablet c4447a4bda0d41f09b88af81b2668b51 (DEFAULT_TABLE table=test-workload [id=5f35bf577cb14bce9aa74c0aa35a2da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:00:24.751339 12561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4447a4bda0d41f09b88af81b2668b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:24.751920 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4447a4bda0d41f09b88af81b2668b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:00:24.762916 12774 tablet_bootstrap.cc:492] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd: Bootstrap starting.
I20260620 23:00:24.766803 12774 tablet_bootstrap.cc:654] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:24.768384 12774 log.cc:824] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:24.775554 12776 tablet_bootstrap.cc:492] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0: Bootstrap starting.
I20260620 23:00:24.783339 12776 tablet_bootstrap.cc:654] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:24.783500 12775 tablet_bootstrap.cc:492] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e: Bootstrap starting.
I20260620 23:00:24.785145 12776 log.cc:824] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:24.786877 12775 tablet_bootstrap.cc:654] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e: Neither blocks nor log segments found. Creating new log.
I20260620 23:00:24.788170 12776 tablet_bootstrap.cc:492] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0: No bootstrap required, opened a new log
I20260620 23:00:24.788173 12774 tablet_bootstrap.cc:492] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd: No bootstrap required, opened a new log
I20260620 23:00:24.788128 12775 log.cc:824] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:00:24.788447 12774 ts_tablet_manager.cc:1397] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.012s
I20260620 23:00:24.788446 12776 ts_tablet_manager.cc:1397] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260620 23:00:24.792452 12495 heartbeater.cc:503] Master 127.10.134.190:43173 was elected leader, sending a full tablet report...
I20260620 23:00:24.799067 12776 raft_consensus.cc:348] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.799489 12776 raft_consensus.cc:374] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:24.799623 12776 raft_consensus.cc:729] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c707c04fc03d49da82f97968100c19d0, State: Initialized, Role: FOLLOWER
I20260620 23:00:24.800522 12776 consensus_queue.cc:260] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.800458 12774 raft_consensus.cc:348] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.800846 12774 raft_consensus.cc:374] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:24.800976 12774 raft_consensus.cc:729] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1834d241b74f0f8152bb84b9e628dd, State: Initialized, Role: FOLLOWER
I20260620 23:00:24.800962 12780 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:24.801294 12780 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.801839 12775 tablet_bootstrap.cc:492] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e: No bootstrap required, opened a new log
I20260620 23:00:24.802201 12775 ts_tablet_manager.cc:1397] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20260620 23:00:24.803282 12780 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:24.804054 12626 heartbeater.cc:503] Master 127.10.134.190:43173 was elected leader, sending a full tablet report...
I20260620 23:00:24.805032 12776 ts_tablet_manager.cc:1428] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0: Time spent starting tablet: real 0.016s user 0.006s sys 0.009s
I20260620 23:00:24.810410 12774 consensus_queue.cc:260] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
W20260620 23:00:24.811043 12514 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.129:33119 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260620 23:00:24.811450 12782 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:24.811708 12782 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.814258 12782 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:24.814319 12775 raft_consensus.cc:348] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.817443 12775 raft_consensus.cc:374] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:00:24.817626 12775 raft_consensus.cc:729] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49a8d52878a43aa83df884ff96e687e, State: Initialized, Role: FOLLOWER
I20260620 23:00:24.821457 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:24.822011 12450 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 0.
I20260620 23:00:24.822268 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:24.822769 12514 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0; no voters:
I20260620 23:00:24.823215 12774 ts_tablet_manager.cc:1428] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd: Time spent starting tablet: real 0.035s user 0.015s sys 0.006s
I20260620 23:00:24.823261 12781 raft_consensus.cc:2781] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:00:24.823403 12781 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:24.823536 12781 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:24.827811 12781 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
W20260620 23:00:24.828433 12514 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497): Timed out: RequestConsensusVote RPC to 127.10.134.131:44497 timed out after 0.006s (SENT)
I20260620 23:00:24.829861 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd"
I20260620 23:00:24.830156 12450 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:24.833019 12775 consensus_queue.cc:260] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.833555 12711 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 0.
W20260620 23:00:24.834043 12383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.130:39755 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
W20260620 23:00:24.834975 12711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.134.130:56547 (request call id 1) took 13 ms (client timeout 6 ms). Trace:
I20260620 23:00:24.836081 12450 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 1.
W20260620 23:00:24.836282 12450 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.134.130:46991 (request call id 2) took 6 ms (client timeout 7 ms). Trace:
W20260620 23:00:24.835184 12711 rpcz_store.cc:269] 0620 23:00:24.821661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:24.821745 (+ 84us) service_pool.cc:227] Handling call
0620 23:00:24.834780 (+ 13035us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x615000072c24. stack: 00007fec1701a66e 00007fec1701a430 00007fec160a15ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fec22542d0a 00007fec260dc5eb 00007fec209c6e06 00007fec209c6b35 00007fec1db569be 00007fec1db5579e 00007fec1db5bb25 00007fec1db5c54f 00007fec1db5c378 00000000009edb7c
0620 23:00:24.834872 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24758656}
I20260620 23:00:24.837177 12788 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 23:00:24.836442 12450 rpcz_store.cc:269] 0620 23:00:24.829638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:24.829704 (+ 66us) service_pool.cc:227] Handling call
0620 23:00:24.836225 (+ 6521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260620 23:00:24.837561 12788 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.838798 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd"
I20260620 23:00:24.838454 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e"
I20260620 23:00:24.839099 12450 raft_consensus.cc:2362] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 1. Re-sending same reply.
I20260620 23:00:24.839547 12514 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0; no voters:
I20260620 23:00:24.839830 12711 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:00:24.839890 12780 raft_consensus.cc:2781] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:00:24.840463 12780 raft_consensus.cc:686] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 1 LEADER]: Becoming Leader. State: Replica: c707c04fc03d49da82f97968100c19d0, State: Running, Role: LEADER
I20260620 23:00:24.840981 12780 consensus_queue.cc:237] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:24.843969 12781 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 1 election: Requested vote from peers 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:24.844290 12711 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 1.
W20260620 23:00:24.844559 12711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.134.130:56547 (request call id 2) took 6 ms (client timeout 7 ms). Trace:
W20260620 23:00:24.844720 12711 rpcz_store.cc:269] 0620 23:00:24.838221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:24.838286 (+ 65us) service_pool.cc:227] Handling call
0620 23:00:24.844507 (+ 6221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1606016}
I20260620 23:00:24.845033 12756 heartbeater.cc:503] Master 127.10.134.190:43173 was elected leader, sending a full tablet report...
I20260620 23:00:24.845774 12775 ts_tablet_manager.cc:1428] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e: Time spent starting tablet: real 0.043s user 0.015s sys 0.020s
I20260620 23:00:24.847056 12308 catalog_manager.cc:5495] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c707c04fc03d49da82f97968100c19d0 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "c707c04fc03d49da82f97968100c19d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:24.848438 12781 consensus_queue.cc:1035] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:24.849040 12781 consensus_queue.cc:1035] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:00:24.849164 12788 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:24.849438 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e"
I20260620 23:00:24.849611 12711 raft_consensus.cc:2362] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 1. Re-sending same reply.
W20260620 23:00:24.860906 12644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260620 23:00:24.865981 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
W20260620 23:00:24.877190 12383 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497): Timed out: RequestConsensusVote RPC to 127.10.134.131:44497 timed out after 0.008s (SENT)
I20260620 23:00:24.877784 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:24.883538 12785 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:00:24.884887 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:24.885983 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:24.887112 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:24.887437 12450 raft_consensus.cc:2379] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e49a8d52878a43aa83df884ff96e687e in current term 1: Already voted for candidate c707c04fc03d49da82f97968100c19d0 in this term.
I20260620 23:00:24.888075 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:24.888994 12788 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 23:00:24.911319 12799 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:00:24.911458 12799 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260620 23:00:25.054759 12671 tablet_service.cc:2970] error setting up scanner aa48422d455a449cacab4671cbae5521 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.143s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.057292 12670 tablet_service.cc:2970] error setting up scanner 0255faec354f4b64be6d06f0f8d64d3e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.145s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
W20260620 23:00:25.073303 12670 tablet_service.cc:2970] error setting up scanner 710fd954213247b3accdc93e9d23a242 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.161s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.075050 12670 tablet_service.cc:2970] error setting up scanner ee4c2ab9269042f798cc1d1c44791da6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.163s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
W20260620 23:00:25.101913 12670 tablet_service.cc:2970] error setting up scanner 428bd82786a74939a9287cd0abc730ec with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.190s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
W20260620 23:00:25.114055 12670 tablet_service.cc:2970] error setting up scanner 377d0ce12e4842fa8c1b277ab60cf1e9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.202s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.168843 12670 tablet_service.cc:2970] error setting up scanner 65fdab58bfa544bea9e980df9cf3477f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.257s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.172928 12670 tablet_service.cc:2970] error setting up scanner bd563855d1f0456d87a85f2bfda49c5e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.261s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
W20260620 23:00:25.275872 12670 tablet_service.cc:2970] error setting up scanner b9841e856c8741d48c9512f2de9ce7c3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.364s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.327788 12670 tablet_service.cc:2970] error setting up scanner 487f2cbe4df84c86a2a6e29f176ce26d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.416s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
W20260620 23:00:25.582615 12670 tablet_service.cc:2970] error setting up scanner fb34c03da9da4265947762dac198d188 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.671s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.586630 12670 tablet_service.cc:2970] error setting up scanner 676101fdf8134fbd9eb82bb282d6a80a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.675s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
W20260620 23:00:25.660851 12711 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:00:25.660998 12711 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260620 23:00:25.661007 12450 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:00:25.661140 12450 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260620 23:00:25.719794 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:25.720068 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:25.720921 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:25.721729 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:25.722195 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:25.722824 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:25.723330 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:25.905748 12670 tablet_service.cc:2970] error setting up scanner c3fa6e04005a4089ad467f6f29381c32 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.994s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:25.941545 12670 tablet_service.cc:2970] error setting up scanner f9771880e9d746c18c4ae58b29d99f42 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.030s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:26.258131 12807 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:26.258405 12807 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:26.259265 12807 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:26.259732 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:26.259771 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:26.260403 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
W20260620 23:00:26.548810 12670 tablet_service.cc:2970] error setting up scanner cc1dbe5f7bfa405ab8ebd5006b3e085f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.288s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:26.715058 12670 tablet_service.cc:2970] error setting up scanner 322663aec6604244a97d516c599668f6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.454s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:26.759052 12807 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:26.761421 12815 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=1, attempt_no=0}) took 1856 ms. Trace:
I20260620 23:00:26.761693 12815 rpcz_store.cc:276] 0620 23:00:24.905390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:24.905440 (+ 50us) service_pool.cc:227] Handling call
0620 23:00:26.761363 (+1855923us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:25.658492 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:25.658625 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:25.658632 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:25.658634 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:25.658731 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:25.658761 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:25.658766 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:25.658823 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:00:25.658825 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:25.658884 (+ 59us) write_op.cc:270] Start()
0620 23:00:26.094063 (+435179us) write_op.cc:276] Timestamp: P: 1781996425659022 usec, L: 0
0620 23:00:26.094069 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:26.094304 (+ 235us) log.cc:842] Serialized 345 byte log entry
0620 23:00:26.760390 (+666086us) op_driver.cc:444] REPLICATION: finished
0620 23:00:26.760854 (+ 464us) write_op.cc:301] APPLY: starting
0620 23:00:26.760873 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:26.760918 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:26.760944 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:26.761040 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:26.761065 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:26.761069 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:00:26.761144 (+ 75us) log.cc:842] Serialized 22 byte log entry
0620 23:00:26.761237 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:26.761254 (+ 17us) write_op.cc:454] Released schema lock
0620 23:00:26.761260 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1970944,"child_traces":[["op",{"apply.queue_time_us":375,"mutex_wait_us":64,"num_ops":1,"prepare.queue_time_us":749347,"prepare.run_cpu_time_us":826,"prepare.run_wall_time_us":435913,"replication_time_us":666298,"spinlock_wait_cycles":896,"thread_start_us":274,"threads_started":1}]]}
I20260620 23:00:27.000280 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:27.000485 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:27.001219 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:27.001693 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:27.001775 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:27.002493 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:27.002772 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:27.529027 12807 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:27.529232 12807 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:27.530085 12807 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:27.530758 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:27.531541 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:27.532199 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:27.532420 12820 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=2, attempt_no=0}) took 1806 ms. Trace:
I20260620 23:00:27.532486 12807 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:27.532538 12820 rpcz_store.cc:276] 0620 23:00:25.725739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:25.725791 (+ 52us) service_pool.cc:227] Handling call
0620 23:00:27.532366 (+1806575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:26.094427 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:26.094562 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:26.094569 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:26.094573 (+ 4us) tablet.cc:660] Decoding operations
0620 23:00:26.094663 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:26.094675 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:26.094682 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:26.094745 (+ 63us) tablet.cc:699] Row locks acquired
0620 23:00:26.094746 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:26.094840 (+ 94us) write_op.cc:270] Start()
0620 23:00:27.032967 (+938127us) write_op.cc:276] Timestamp: P: 1781996426094955 usec, L: 0
0620 23:00:27.032971 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:00:27.033208 (+ 237us) log.cc:842] Serialized 354 byte log entry
0620 23:00:27.530205 (+496997us) op_driver.cc:444] REPLICATION: finished
0620 23:00:27.531691 (+ 1486us) write_op.cc:301] APPLY: starting
0620 23:00:27.531728 (+ 37us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:27.531762 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:27.531798 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:27.531860 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:27.531883 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:27.531887 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:00:27.531977 (+ 90us) log.cc:842] Serialized 22 byte log entry
0620 23:00:27.532257 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:27.532272 (+ 15us) write_op.cc:454] Released schema lock
0620 23:00:27.532279 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1435,"num_ops":1,"prepare.queue_time_us":368004,"prepare.run_cpu_time_us":847,"prepare.run_wall_time_us":938901,"raft.queue_time_us":326,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":250,"replication_time_us":497206,"spinlock_wait_cycles":3712,"thread_start_us":1250,"threads_started":1}]]}
I20260620 23:00:27.682628 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:27.682819 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:27.683465 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:27.684172 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:27.684247 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:27.684896 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:27.685187 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:27.989216 12807 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:27.989396 12807 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:27.990160 12807 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:27.990697 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:27.990774 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:27.991431 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:27.992182 12807 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:28.062599 12807 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:28.062788 12807 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:28.063581 12807 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:28.064025 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:28.064107 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:28.064697 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:28.065150 12807 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:28.133988 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:28.134209 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:28.134958 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:28.135341 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:28.135432 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:28.136056 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
W20260620 23:00:28.471729 12670 tablet_service.cc:2970] error setting up scanner 8fa38540ed434745b77b274b3b519f75 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.407s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:28.691809 12824 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:28.692060 12824 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:28.692914 12824 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:28.693459 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:28.693591 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:28.694329 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:28.694789 12824 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:29.022557 12670 tablet_service.cc:2970] error setting up scanner a84af4b4cf8341ceb3046d5f546bfe69 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.329s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:29.509738 12450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 836) took 1374 ms. Trace:
I20260620 23:00:29.510160 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:29.509924 12450 rpcz_store.cc:276] 0620 23:00:28.134953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:28.135231 (+ 278us) service_pool.cc:227] Handling call
0620 23:00:28.135457 (+ 226us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:28.135586 (+ 129us) raft_consensus.cc:1520] Early marking committed up to index 0
0620 23:00:28.135590 (+ 4us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:28.135758 (+ 168us) write_op.cc:270] Start()
0620 23:00:28.785911 (+650153us) write_op.cc:276] Timestamp: P: 1781996427680817 usec, L: 0
0620 23:00:28.786567 (+ 656us) write_op.cc:270] Start()
0620 23:00:29.507733 (+721166us) write_op.cc:276] Timestamp: P: 1781996427918643 usec, L: 0
0620 23:00:29.508476 (+ 743us) log.cc:842] Serialized 708 byte log entry
0620 23:00:29.508831 (+ 355us) raft_consensus.cc:1631] Marking committed up to 7
0620 23:00:29.509558 (+ 727us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:29.509602 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:29.509613 (+ 11us) raft_consensus.cc:1671] finished
0620 23:00:29.509617 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:29.509678 (+ 61us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:28.786479 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:28.786590 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:28.786595 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:28.786596 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:28.786700 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:28.786721 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:28.786725 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:28.786781 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:00:28.786786 (+ 5us) write_op.cc:260] PREPARE: finished
0620 23:00:29.508862 (+722076us) op_driver.cc:444] REPLICATION: finished
0620 23:00:29.509282 (+ 420us) write_op.cc:301] APPLY: starting
0620 23:00:29.509297 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:29.509340 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:29.509344 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:29.509495 (+ 151us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:29.509538 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:29.509602 (+ 64us) write_op.cc:312] APPLY: finished
0620 23:00:29.509703 (+ 101us) log.cc:842] Serialized 22 byte log entry
0620 23:00:29.509806 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:29.509821 (+ 15us) write_op.cc:454] Released schema lock
0620 23:00:29.509828 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0620 23:00:29.509922 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:29.509927 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:00:29.508273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:29.508400 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:29.508406 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:29.508408 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:29.508504 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:29.508549 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:29.508558 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:29.508626 (+ 68us) tablet.cc:699] Row locks acquired
0620 23:00:29.508631 (+ 5us) write_op.cc:260] PREPARE: finished
0620 23:00:29.509251 (+ 620us) op_driver.cc:444] REPLICATION: finished
0620 23:00:29.509702 (+ 451us) write_op.cc:301] APPLY: starting
0620 23:00:29.509722 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:29.509868 (+ 146us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:29.509872 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:29.510008 (+ 136us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:29.510036 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:29.510086 (+ 50us) write_op.cc:312] APPLY: finished
0620 23:00:29.510161 (+ 75us) log.cc:842] Serialized 22 byte log entry
0620 23:00:29.510256 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:29.510270 (+ 14us) write_op.cc:454] Released schema lock
0620 23:00:29.510275 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0620 23:00:29.510557 (+ 282us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:29.510563 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":253,"threads_started":1,"wal-append.queue_time_us":346,"child_traces":[["op",{"apply.queue_time_us":373,"apply.run_cpu_time_us":725,"apply.run_wall_time_us":725,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":389,"prepare.run_cpu_time_us":402,"prepare.run_wall_time_us":400,"replication_time_us":1373093,"thread_start_us":526,"threads_started":2}],["op",{"apply.queue_time_us":391,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":926,"num_ops":1,"partition_lock_wait_us":34,"prepare.queue_time_us":416,"prepare.run_cpu_time_us":498,"prepare.run_wall_time_us":498,"replication_time_us":722626,"thread_start_us":559,"threads_started":2}]]}
I20260620 23:00:30.189800 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:30.190083 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:30.191043 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:30.191584 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:30.191608 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:30.192490 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:30.192735 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:30.194106 12840 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=7, attempt_no=0}) took 2128 ms. Trace:
I20260620 23:00:30.194209 12840 rpcz_store.cc:276] 0620 23:00:28.065637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:28.065711 (+ 74us) service_pool.cc:227] Handling call
0620 23:00:30.194063 (+2128352us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:28.594038 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:28.594177 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:28.594183 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:28.594185 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:28.594282 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:28.594297 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:28.594304 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:28.594358 (+ 54us) tablet.cc:699] Row locks acquired
0620 23:00:28.594360 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:28.594417 (+ 57us) write_op.cc:270] Start()
0620 23:00:29.488573 (+894156us) write_op.cc:276] Timestamp: P: 1781996428594530 usec, L: 0
0620 23:00:29.488586 (+ 13us) op_driver.cc:331] REPLICATION: starting
0620 23:00:29.488826 (+ 240us) log.cc:842] Serialized 354 byte log entry
0620 23:00:30.192943 (+704117us) op_driver.cc:444] REPLICATION: finished
0620 23:00:30.193383 (+ 440us) write_op.cc:301] APPLY: starting
0620 23:00:30.193402 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:30.193445 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:30.193482 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:30.193576 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:30.193627 (+ 51us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:30.193637 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:00:30.193743 (+ 106us) log.cc:842] Serialized 22 byte log entry
0620 23:00:30.193906 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:30.193926 (+ 20us) write_op.cc:454] Released schema lock
0620 23:00:30.193957 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":370,"num_ops":1,"prepare.queue_time_us":527854,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":894871,"replication_time_us":704328,"thread_start_us":252,"threads_started":1}]]}
I20260620 23:00:30.207549 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:30.207746 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:30.208593 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:30.208966 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:30.209077 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:30.209897 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:30.333266 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:30.335177 12840 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=8, attempt_no=0}) took 1639 ms. Trace:
I20260620 23:00:30.335294 12840 rpcz_store.cc:276] 0620 23:00:28.696182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:28.696238 (+ 56us) service_pool.cc:227] Handling call
0620 23:00:30.335122 (+1638884us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:29.488952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:29.489089 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:29.489096 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:29.489097 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:29.489235 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:29.489247 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:29.489270 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:29.489335 (+ 65us) tablet.cc:699] Row locks acquired
0620 23:00:29.489337 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:29.489408 (+ 71us) write_op.cc:270] Start()
0620 23:00:29.667588 (+178180us) write_op.cc:276] Timestamp: P: 1781996429489459 usec, L: 0
0620 23:00:29.667600 (+ 12us) op_driver.cc:331] REPLICATION: starting
0620 23:00:29.667858 (+ 258us) log.cc:842] Serialized 354 byte log entry
0620 23:00:30.334454 (+666596us) op_driver.cc:444] REPLICATION: finished
0620 23:00:30.334564 (+ 110us) write_op.cc:301] APPLY: starting
0620 23:00:30.334588 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:30.334661 (+ 73us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:30.334701 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:30.334781 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:30.334820 (+ 39us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:30.334831 (+ 11us) write_op.cc:312] APPLY: finished
0620 23:00:30.334920 (+ 89us) log.cc:842] Serialized 22 byte log entry
0620 23:00:30.335000 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:30.335018 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:30.335027 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"num_ops":1,"prepare.queue_time_us":792222,"prepare.run_cpu_time_us":908,"prepare.run_wall_time_us":178984,"replication_time_us":666827}]]}
I20260620 23:00:30.490676 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:30.490900 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:30.491683 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:30.492250 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:30.492345 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:30.493012 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:30.493427 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:31.098189 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:31.098413 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:31.099234 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:31.099820 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:31.100080 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:31.100908 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:31.101179 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:31.281222 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:31.281426 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:31.282182 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:31.282711 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:31.282820 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:31.283416 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:31.368395 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:31.368633 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:31.369370 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:31.369917 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:31.370023 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:31.370601 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:31.370955 12846 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=10, attempt_no=0}) took 1033 ms. Trace:
I20260620 23:00:31.371363 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:31.371070 12846 rpcz_store.cc:276] 0620 23:00:30.337250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:30.337320 (+ 70us) service_pool.cc:227] Handling call
0620 23:00:31.370892 (+1033572us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:30.635765 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:30.635877 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:30.635882 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:30.635884 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:30.635956 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:30.635965 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:30.635968 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:30.636012 (+ 44us) tablet.cc:699] Row locks acquired
0620 23:00:30.636013 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:30.636051 (+ 38us) write_op.cc:270] Start()
0620 23:00:31.227200 (+591149us) write_op.cc:276] Timestamp: P: 1781996430636174 usec, L: 0
0620 23:00:31.227205 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:31.227463 (+ 258us) log.cc:842] Serialized 354 byte log entry
0620 23:00:31.369925 (+142462us) op_driver.cc:444] REPLICATION: finished
0620 23:00:31.370361 (+ 436us) write_op.cc:301] APPLY: starting
0620 23:00:31.370379 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:31.370422 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:31.370445 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:31.370520 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:31.370556 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:31.370562 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:31.370652 (+ 90us) log.cc:842] Serialized 22 byte log entry
0620 23:00:31.370772 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:31.370790 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:31.370799 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":352,"num_ops":1,"prepare.queue_time_us":298068,"prepare.run_cpu_time_us":721,"prepare.run_wall_time_us":591793,"raft.queue_time_us":124,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":142688,"spinlock_wait_cycles":91904}]]}
W20260620 23:00:31.546686 12670 tablet_service.cc:2970] error setting up scanner 0ffbdb60a0634972b0130c864b449d06 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.264s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:32.202500 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:32.369647 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:32.369861 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:32.370766 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:32.371336 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:32.371335 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:32.372038 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:32.372524 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:32.372785 12854 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=11, attempt_no=0}) took 1271 ms. Trace:
I20260620 23:00:32.372910 12854 rpcz_store.cc:276] 0620 23:00:31.101717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:31.101790 (+ 73us) service_pool.cc:227] Handling call
0620 23:00:32.372728 (+1270938us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:31.227582 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:31.227701 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:31.227707 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:31.227710 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:31.227799 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:31.227810 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:31.227815 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:31.227876 (+ 61us) tablet.cc:699] Row locks acquired
0620 23:00:31.227879 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:31.227918 (+ 39us) write_op.cc:270] Start()
0620 23:00:31.737107 (+509189us) write_op.cc:276] Timestamp: P: 1781996431228081 usec, L: 0
0620 23:00:31.737113 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:31.737344 (+ 231us) log.cc:842] Serialized 354 byte log entry
0620 23:00:32.371242 (+633898us) op_driver.cc:444] REPLICATION: finished
0620 23:00:32.372135 (+ 893us) write_op.cc:301] APPLY: starting
0620 23:00:32.372155 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:32.372199 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:32.372237 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:32.372335 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:32.372415 (+ 80us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:32.372425 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:00:32.372537 (+ 112us) log.cc:842] Serialized 22 byte log entry
0620 23:00:32.372614 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:32.372632 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:32.372638 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":821,"num_ops":1,"prepare.queue_time_us":125302,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":509834,"replication_time_us":634103,"thread_start_us":832,"threads_started":1}]]}
I20260620 23:00:32.397491 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:32.397684 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:32.398564 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:32.398942 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:32.399494 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:32.400070 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:32.400314 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:33.377765 12670 tablet_service.cc:2970] error setting up scanner 7d749a634dc84653a9e35f0835e1ca75 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.979s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:33.406606 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:33.406838 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:33.407527 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:33.408047 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:33.408120 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:33.408893 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:33.409179 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:33.409688 12861 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=12, attempt_no=0}) took 2036 ms. Trace:
I20260620 23:00:33.409811 12861 rpcz_store.cc:276] 0620 23:00:31.373012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:31.373060 (+ 48us) service_pool.cc:227] Handling call
0620 23:00:33.409642 (+2036582us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:31.737460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:31.737585 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:31.737591 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:31.737595 (+ 4us) tablet.cc:660] Decoding operations
0620 23:00:31.737681 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:31.737691 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:31.737694 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:31.737745 (+ 51us) tablet.cc:699] Row locks acquired
0620 23:00:31.737747 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:31.737816 (+ 69us) write_op.cc:270] Start()
0620 23:00:32.727939 (+990123us) write_op.cc:276] Timestamp: P: 1781996431737969 usec, L: 0
0620 23:00:32.727942 (+ 3us) op_driver.cc:331] REPLICATION: starting
0620 23:00:32.728127 (+ 185us) log.cc:842] Serialized 354 byte log entry
0620 23:00:33.408498 (+680371us) op_driver.cc:444] REPLICATION: finished
0620 23:00:33.409108 (+ 610us) write_op.cc:301] APPLY: starting
0620 23:00:33.409127 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:33.409173 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:33.409210 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:33.409295 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:33.409336 (+ 41us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:33.409345 (+ 9us) write_op.cc:312] APPLY: finished
0620 23:00:33.409452 (+ 107us) log.cc:842] Serialized 22 byte log entry
0620 23:00:33.409519 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:33.409534 (+ 15us) write_op.cc:454] Released schema lock
0620 23:00:33.409540 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":544,"num_ops":1,"prepare.queue_time_us":363936,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":990772,"raft.queue_time_us":53,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"replication_time_us":680529,"thread_start_us":1305,"threads_started":1}]]}
I20260620 23:00:33.537515 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:33.537708 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:33.538491 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:33.538980 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:33.539072 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:33.539750 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:34.203030 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:34.205628 12869 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=13, attempt_no=0}) took 1830 ms. Trace:
I20260620 23:00:34.205765 12869 rpcz_store.cc:276] 0620 23:00:32.374678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:32.374748 (+ 70us) service_pool.cc:227] Handling call
0620 23:00:34.205562 (+1830814us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:32.728272 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:32.728419 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:32.728425 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:32.728428 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:32.728531 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:32.728540 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:32.728543 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:32.728612 (+ 69us) tablet.cc:699] Row locks acquired
0620 23:00:32.728614 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:32.728670 (+ 56us) write_op.cc:270] Start()
0620 23:00:33.247842 (+519172us) write_op.cc:276] Timestamp: P: 1781996432728806 usec, L: 0
0620 23:00:33.247848 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:33.248074 (+ 226us) log.cc:842] Serialized 354 byte log entry
0620 23:00:34.204482 (+956408us) op_driver.cc:444] REPLICATION: finished
0620 23:00:34.204966 (+ 484us) write_op.cc:301] APPLY: starting
0620 23:00:34.204987 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:34.205029 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:34.205068 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:34.205167 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:34.205202 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:34.205208 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:34.205308 (+ 100us) log.cc:842] Serialized 22 byte log entry
0620 23:00:34.205410 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:34.205429 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:34.205439 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":410,"num_ops":1,"prepare.queue_time_us":353079,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":519872,"replication_time_us":956610,"thread_start_us":338,"threads_started":1}]]}
I20260620 23:00:34.238215 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:34.238430 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:34.239180 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:34.239620 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:34.239737 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:34.240352 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:34.528417 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:34.530494 12869 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=14, attempt_no=0}) took 1118 ms. Trace:
I20260620 23:00:34.530658 12869 rpcz_store.cc:276] 0620 23:00:33.411766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:33.411840 (+ 74us) service_pool.cc:227] Handling call
0620 23:00:34.530405 (+1118565us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:33.412274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:33.412380 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:33.412384 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:00:33.412385 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:33.412453 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:33.412460 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:33.412463 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:33.412502 (+ 39us) tablet.cc:699] Row locks acquired
0620 23:00:33.412504 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:33.412531 (+ 27us) write_op.cc:270] Start()
0620 23:00:33.543685 (+131154us) write_op.cc:276] Timestamp: P: 1781996433412483 usec, L: 0
0620 23:00:33.543690 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:33.543865 (+ 175us) log.cc:842] Serialized 354 byte log entry
0620 23:00:34.529504 (+985639us) op_driver.cc:444] REPLICATION: finished
0620 23:00:34.529647 (+ 143us) write_op.cc:301] APPLY: starting
0620 23:00:34.529666 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:34.529710 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:34.529754 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:34.530006 (+ 252us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:34.530048 (+ 42us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:34.530056 (+ 8us) write_op.cc:312] APPLY: finished
0620 23:00:34.530147 (+ 91us) log.cc:842] Serialized 22 byte log entry
0620 23:00:34.530244 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:34.530268 (+ 24us) write_op.cc:454] Released schema lock
0620 23:00:34.530279 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":77,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":607,"prepare.run_wall_time_us":131691,"replication_time_us":985786}]]}
I20260620 23:00:34.728874 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:34.729074 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:34.729868 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:34.730381 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:34.730480 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:34.731144 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:34.967214 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:35.239959 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:35.240166 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:35.240984 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:35.241479 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:35.241604 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:35.242290 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
W20260620 23:00:35.389746 12670 tablet_service.cc:2970] error setting up scanner a9f7bb4dd0264133885e84db8c278950 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.418s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:36.046198 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:36.046523 12711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:56547 (request call id 2569) took 1001 ms. Trace:
I20260620 23:00:36.046449 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:36.046718 12711 rpcz_store.cc:276] 0620 23:00:35.045426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:35.045503 (+ 77us) service_pool.cc:227] Handling call
0620 23:00:35.045638 (+ 135us) raft_consensus.cc:1457] Updating replica for 1 ops
0620 23:00:35.045704 (+ 66us) raft_consensus.cc:1520] Early marking committed up to index 0
0620 23:00:35.045706 (+ 2us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0620 23:00:35.045806 (+ 100us) write_op.cc:270] Start()
0620 23:00:36.044980 (+999174us) write_op.cc:276] Timestamp: P: 1781996434971951 usec, L: 0
0620 23:00:36.045681 (+ 701us) log.cc:842] Serialized 354 byte log entry
0620 23:00:36.046041 (+ 360us) raft_consensus.cc:1631] Marking committed up to 18
0620 23:00:36.046052 (+ 11us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:36.046089 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:36.046414 (+ 325us) raft_consensus.cc:1671] finished
0620 23:00:36.046416 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:36.046480 (+ 64us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:36.045490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:36.045640 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:36.045645 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:36.045647 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:36.045724 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:36.045739 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:36.045744 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:36.045792 (+ 48us) tablet.cc:699] Row locks acquired
0620 23:00:36.045793 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":271,"threads_started":1,"wal-append.queue_time_us":346,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":342,"prepare.run_cpu_time_us":419,"prepare.run_wall_time_us":419,"thread_start_us":222,"threads_started":1}]]}
I20260620 23:00:36.047501 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:36.048162 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:36.048493 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:36.049103 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:36.049409 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:36.050443 12885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=17, attempt_no=0}) took 1079 ms. Trace:
I20260620 23:00:36.050637 12885 rpcz_store.cc:276] 0620 23:00:34.971018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:34.971071 (+ 53us) service_pool.cc:227] Handling call
0620 23:00:36.050389 (+1079318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:34.971616 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:34.971736 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:34.971741 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:34.971744 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:34.971832 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:34.971845 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:34.971853 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:34.971899 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:00:34.971902 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:34.971942 (+ 40us) write_op.cc:270] Start()
0620 23:00:35.044097 (+ 72155us) write_op.cc:276] Timestamp: P: 1781996434971951 usec, L: 0
0620 23:00:35.044104 (+ 7us) op_driver.cc:331] REPLICATION: starting
0620 23:00:35.044301 (+ 197us) log.cc:842] Serialized 354 byte log entry
0620 23:00:36.049310 (+1005009us) op_driver.cc:444] REPLICATION: finished
0620 23:00:36.049841 (+ 531us) write_op.cc:301] APPLY: starting
0620 23:00:36.049868 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:36.049923 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:36.050004 (+ 81us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:36.050097 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:36.050126 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:36.050131 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:00:36.050204 (+ 73us) log.cc:842] Serialized 22 byte log entry
0620 23:00:36.050285 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:36.050302 (+ 17us) write_op.cc:454] Released schema lock
0620 23:00:36.050308 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":466,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":706,"prepare.run_wall_time_us":72779,"replication_time_us":1005177,"thread_start_us":313,"threads_started":1}]]}
I20260620 23:00:36.390432 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:36.390650 12450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 1693) took 1148 ms. Trace:
I20260620 23:00:36.390816 12450 rpcz_store.cc:276] 0620 23:00:35.241859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:35.242101 (+ 242us) service_pool.cc:227] Handling call
0620 23:00:35.242332 (+ 231us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:35.242455 (+ 123us) raft_consensus.cc:1520] Early marking committed up to index 17
0620 23:00:35.242751 (+ 296us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:35.242913 (+ 162us) write_op.cc:270] Start()
0620 23:00:35.447115 (+204202us) write_op.cc:276] Timestamp: P: 1781996434533369 usec, L: 0
0620 23:00:35.447450 (+ 335us) write_op.cc:270] Start()
0620 23:00:36.388635 (+941185us) write_op.cc:276] Timestamp: P: 1781996434971951 usec, L: 0
0620 23:00:36.389587 (+ 952us) log.cc:842] Serialized 708 byte log entry
0620 23:00:36.389959 (+ 372us) raft_consensus.cc:1631] Marking committed up to 18
0620 23:00:36.390296 (+ 337us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:36.390336 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:36.390520 (+ 184us) raft_consensus.cc:1671] finished
0620 23:00:36.390522 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:36.390581 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:35.447424 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:35.447581 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:35.447587 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:35.447594 (+ 7us) tablet.cc:660] Decoding operations
0620 23:00:35.447678 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:35.447693 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:35.447696 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:35.447746 (+ 50us) tablet.cc:699] Row locks acquired
0620 23:00:35.447748 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:36.390001 (+942253us) op_driver.cc:444] REPLICATION: finished
0620 23:00:36.390373 (+ 372us) write_op.cc:301] APPLY: starting
0620 23:00:36.390389 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:36.390435 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:36.390437 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:36.390531 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:36.390569 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:36.390606 (+ 37us) write_op.cc:312] APPLY: finished
0620 23:00:36.390836 (+ 230us) log.cc:842] Serialized 22 byte log entry
0620 23:00:36.390910 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:36.390922 (+ 12us) write_op.cc:454] Released schema lock
0620 23:00:36.390927 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0620 23:00:36.390986 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:36.390988 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:00:36.389209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:36.389554 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:36.389561 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:36.389566 (+ 5us) tablet.cc:660] Decoding operations
0620 23:00:36.389677 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:36.389690 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:36.389693 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:36.389742 (+ 49us) tablet.cc:699] Row locks acquired
0620 23:00:36.389743 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":251,"threads_started":1,"wal-append.queue_time_us":351,"child_traces":[["op",{"apply.queue_time_us":297,"apply.run_cpu_time_us":568,"apply.run_wall_time_us":664,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":101,"prepare.run_cpu_time_us":425,"prepare.run_wall_time_us":426,"replication_time_us":1147073,"thread_start_us":207,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":436,"prepare.run_cpu_time_us":625,"prepare.run_wall_time_us":625,"thread_start_us":275,"threads_started":1}]]}
I20260620 23:00:36.856870 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:36.857098 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:36.857955 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:36.858639 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:36.858757 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:36.859510 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:36.859969 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:36.860212 12892 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=18, attempt_no=0}) took 1888 ms. Trace:
I20260620 23:00:36.860333 12892 rpcz_store.cc:276] 0620 23:00:34.971753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:34.971794 (+ 41us) service_pool.cc:227] Handling call
0620 23:00:36.860152 (+1888358us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:35.044431 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:35.044572 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:35.044579 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:35.044581 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:35.044675 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:35.044688 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:35.044692 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:35.044751 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:00:35.044754 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:35.044803 (+ 49us) write_op.cc:270] Start()
0620 23:00:35.742987 (+698184us) write_op.cc:276] Timestamp: P: 1781996435044842 usec, L: 0
0620 23:00:35.742992 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:35.743236 (+ 244us) log.cc:842] Serialized 354 byte log entry
0620 23:00:36.858376 (+1115140us) op_driver.cc:444] REPLICATION: finished
0620 23:00:36.859188 (+ 812us) write_op.cc:301] APPLY: starting
0620 23:00:36.859207 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:36.859250 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:36.859306 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:36.859405 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:36.859441 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:36.859448 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:00:36.859553 (+ 105us) log.cc:842] Serialized 22 byte log entry
0620 23:00:36.859875 (+ 322us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:36.859894 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:36.859901 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":741,"num_ops":1,"prepare.queue_time_us":72315,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":698908,"replication_time_us":1115360,"thread_start_us":648,"threads_started":1}]]}
W20260620 23:00:36.965026 12670 tablet_service.cc:2970] error setting up scanner 65c75c0807be48a8b9f9a4d3806fc0f4 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.106s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:37.077252 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:37.077507 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:37.078388 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:37.078855 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:37.079001 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:37.079885 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:37.801357 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:37.801574 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:37.802425 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:37.802870 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:37.802966 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:37.803532 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:37.803867 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:37.804808 12899 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=19, attempt_no=0}) took 1752 ms. Trace:
I20260620 23:00:37.804983 12899 rpcz_store.cc:276] 0620 23:00:36.052430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:36.052496 (+ 66us) service_pool.cc:227] Handling call
0620 23:00:37.804760 (+1752264us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:36.053049 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:36.053146 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:36.053150 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:00:36.053151 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:36.053223 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:36.053232 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:36.053235 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:36.053277 (+ 42us) tablet.cc:699] Row locks acquired
0620 23:00:36.053278 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:36.053312 (+ 34us) write_op.cc:270] Start()
0620 23:00:36.883476 (+830164us) write_op.cc:276] Timestamp: P: 1781996436053343 usec, L: 0
0620 23:00:36.883482 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:36.883715 (+ 233us) log.cc:842] Serialized 354 byte log entry
0620 23:00:37.803614 (+919899us) op_driver.cc:444] REPLICATION: finished
0620 23:00:37.804287 (+ 673us) write_op.cc:301] APPLY: starting
0620 23:00:37.804302 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:37.804337 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:37.804355 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:37.804416 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:37.804441 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:37.804445 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:00:37.804576 (+ 131us) log.cc:842] Serialized 22 byte log entry
0620 23:00:37.804666 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:37.804681 (+ 15us) write_op.cc:454] Released schema lock
0620 23:00:37.804687 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":611,"num_ops":1,"prepare.queue_time_us":72,"prepare.run_cpu_time_us":686,"prepare.run_wall_time_us":830757,"replication_time_us":920107,"thread_start_us":499,"threads_started":1}]]}
I20260620 23:00:38.279026 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:38.279234 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:38.280074 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:38.280537 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:38.280597 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:38.280814 12899 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=20, attempt_no=0}) took 1418 ms. Trace:
I20260620 23:00:38.280951 12899 rpcz_store.cc:276] 0620 23:00:36.862198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:36.862266 (+ 68us) service_pool.cc:227] Handling call
0620 23:00:38.280753 (+1418487us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:36.883840 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:36.883969 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:36.883976 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:36.883981 (+ 5us) tablet.cc:660] Decoding operations
0620 23:00:36.884074 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:36.884086 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:36.884091 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:36.884172 (+ 81us) tablet.cc:699] Row locks acquired
0620 23:00:36.884175 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:36.884220 (+ 45us) write_op.cc:270] Start()
0620 23:00:36.926393 (+ 42173us) write_op.cc:276] Timestamp: P: 1781996436884380 usec, L: 0
0620 23:00:36.926398 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:36.926647 (+ 249us) log.cc:842] Serialized 354 byte log entry
0620 23:00:38.280080 (+1353433us) op_driver.cc:444] REPLICATION: finished
0620 23:00:38.280205 (+ 125us) write_op.cc:301] APPLY: starting
0620 23:00:38.280227 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:38.280271 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:38.280300 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:38.280375 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:38.280430 (+ 55us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:38.280436 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:38.280528 (+ 92us) log.cc:842] Serialized 22 byte log entry
0620 23:00:38.280606 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:38.280624 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:38.280635 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72,"num_ops":1,"prepare.queue_time_us":21213,"prepare.run_cpu_time_us":841,"prepare.run_wall_time_us":42912,"replication_time_us":1353653}]]}
I20260620 23:00:38.281490 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:38.439034 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:38.439515 12450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 1695) took 1360 ms. Trace:
I20260620 23:00:38.439636 12450 rpcz_store.cc:276] 0620 23:00:37.078944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:37.079020 (+ 76us) service_pool.cc:227] Handling call
0620 23:00:37.079270 (+ 250us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:37.079386 (+ 116us) raft_consensus.cc:1520] Early marking committed up to index 20
0620 23:00:37.079814 (+ 428us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:37.079941 (+ 127us) write_op.cc:270] Start()
0620 23:00:37.825111 (+745170us) write_op.cc:276] Timestamp: P: 1781996436053343 usec, L: 0
0620 23:00:37.825669 (+ 558us) write_op.cc:270] Start()
0620 23:00:38.437838 (+612169us) write_op.cc:276] Timestamp: P: 1781996436884380 usec, L: 0
0620 23:00:38.438504 (+ 666us) log.cc:842] Serialized 708 byte log entry
0620 23:00:38.438845 (+ 341us) raft_consensus.cc:1631] Marking committed up to 20
0620 23:00:38.438859 (+ 14us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:38.438900 (+ 41us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:38.439418 (+ 518us) raft_consensus.cc:1671] finished
0620 23:00:38.439421 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:38.439480 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:37.825598 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:37.825714 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:37.825719 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:37.825720 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:37.825799 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:37.825812 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:37.825815 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:37.825859 (+ 44us) tablet.cc:699] Row locks acquired
0620 23:00:37.825861 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:00:38.438377 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:38.438487 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:38.438493 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:38.438495 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:38.438574 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:38.438586 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:38.438589 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:38.438635 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:00:38.438636 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":262,"threads_started":1,"wal-append.queue_time_us":352,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":355,"prepare.run_cpu_time_us":407,"prepare.run_wall_time_us":407,"thread_start_us":235,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":348,"prepare.run_cpu_time_us":345,"prepare.run_wall_time_us":346,"thread_start_us":222,"threads_started":1}]]}
W20260620 23:00:38.720826 12670 tablet_service.cc:2970] error setting up scanner 3efde98b79914413af7ce1a7a738d18f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.440s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:39.268304 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:39.272298 12910 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=21, attempt_no=0}) took 1465 ms. Trace:
I20260620 23:00:39.272493 12910 rpcz_store.cc:276] 0620 23:00:37.806782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:37.806854 (+ 72us) service_pool.cc:227] Handling call
0620 23:00:39.272232 (+1465378us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:37.807540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:37.807641 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:37.807645 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:00:37.807646 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:37.807725 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:37.807733 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:37.807736 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:37.807780 (+ 44us) tablet.cc:699] Row locks acquired
0620 23:00:37.807781 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:37.807812 (+ 31us) write_op.cc:270] Start()
0620 23:00:37.966037 (+158225us) write_op.cc:276] Timestamp: P: 1781996437807941 usec, L: 0
0620 23:00:37.966042 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:37.966288 (+ 246us) log.cc:842] Serialized 354 byte log entry
0620 23:00:39.271200 (+1304912us) op_driver.cc:444] REPLICATION: finished
0620 23:00:39.271632 (+ 432us) write_op.cc:301] APPLY: starting
0620 23:00:39.271666 (+ 34us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:39.271711 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:39.271773 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:39.271887 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:39.271920 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:39.271926 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:39.272024 (+ 98us) log.cc:842] Serialized 22 byte log entry
0620 23:00:39.272109 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:39.272127 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:39.272135 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":370,"num_ops":1,"prepare.queue_time_us":291,"prepare.run_cpu_time_us":775,"prepare.run_wall_time_us":158846,"replication_time_us":1305134,"thread_start_us":439,"threads_started":2}]]}
I20260620 23:00:39.298544 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:39.298795 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:39.299577 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:39.300112 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:39.300237 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:39.300793 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:39.728863 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:39.730686 12910 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=22, attempt_no=0}) took 1447 ms. Trace:
I20260620 23:00:39.730924 12910 rpcz_store.cc:276] 0620 23:00:38.282742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:38.282809 (+ 67us) service_pool.cc:227] Handling call
0620 23:00:39.730598 (+1447789us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:38.283233 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:38.283326 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:38.283330 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:00:38.283331 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:38.283398 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:38.283405 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:38.283408 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:38.283451 (+ 43us) tablet.cc:699] Row locks acquired
0620 23:00:38.283452 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:38.283485 (+ 33us) write_op.cc:270] Start()
0620 23:00:39.185666 (+902181us) write_op.cc:276] Timestamp: P: 1781996438283574 usec, L: 0
0620 23:00:39.185674 (+ 8us) op_driver.cc:331] REPLICATION: starting
0620 23:00:39.185904 (+ 230us) log.cc:842] Serialized 354 byte log entry
0620 23:00:39.729908 (+544004us) op_driver.cc:444] REPLICATION: finished
0620 23:00:39.730051 (+ 143us) write_op.cc:301] APPLY: starting
0620 23:00:39.730072 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:39.730119 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:39.730158 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:39.730245 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:39.730286 (+ 41us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:39.730293 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:00:39.730381 (+ 88us) log.cc:842] Serialized 22 byte log entry
0620 23:00:39.730477 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:39.730496 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:39.730502 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":70,"num_ops":1,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":728,"prepare.run_wall_time_us":902796,"replication_time_us":544200}]]}
I20260620 23:00:39.998926 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:39.999132 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:40.000214 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:40.000780 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:40.000805 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:40.001457 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:40.155808 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:40.156039 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:40.156980 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:40.157639 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:40.157804 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:40.158452 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:40.158746 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:40.485246 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:41.113897 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:41.114143 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:41.114941 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:41.115751 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:41.115742 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:41.116477 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:41.116889 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:41.117626 12920 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=24, attempt_no=0}) took 1384 ms. Trace:
I20260620 23:00:41.117982 12920 rpcz_store.cc:276] 0620 23:00:39.732681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:39.732811 (+ 130us) service_pool.cc:227] Handling call
0620 23:00:41.117527 (+1384716us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:39.733486 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:39.733607 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:39.733622 (+ 15us) write_op.cc:435] Acquired schema lock
0620 23:00:39.733624 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:39.733725 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:39.733733 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:39.733736 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:39.733775 (+ 39us) tablet.cc:699] Row locks acquired
0620 23:00:39.733776 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:39.733828 (+ 52us) write_op.cc:270] Start()
0620 23:00:40.580969 (+847141us) write_op.cc:276] Timestamp: P: 1781996439733894 usec, L: 0
0620 23:00:40.580974 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:40.581199 (+ 225us) log.cc:842] Serialized 354 byte log entry
0620 23:00:41.116374 (+535175us) op_driver.cc:444] REPLICATION: finished
0620 23:00:41.116960 (+ 586us) write_op.cc:301] APPLY: starting
0620 23:00:41.116980 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:41.117025 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:41.117052 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:41.117147 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:41.117190 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:41.117200 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:00:41.117306 (+ 106us) log.cc:842] Serialized 22 byte log entry
0620 23:00:41.117390 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:41.117409 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:41.117417 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":522,"num_ops":1,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":847839,"raft.queue_time_us":511,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":327,"replication_time_us":535373,"spinlock_wait_cycles":214784,"thread_start_us":413,"threads_started":1}]]}
I20260620 23:00:41.433202 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:41.433431 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:41.434379 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:41.434901 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:41.434908 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:41.435778 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:41.436041 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:41.515594 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:41.515776 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:41.516395 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:41.516831 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:41.517315 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:41.517904 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:41.518208 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:41.576148 12670 tablet_service.cc:2970] error setting up scanner 6a2e003c8a8e48eeaf18afcb512db8d9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.459s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:41.977072 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:41.977339 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:41.978143 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:41.978698 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:41.978753 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:41.979429 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:41.979974 12927 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=25, attempt_no=0}) took 1820 ms. Trace:
I20260620 23:00:41.980101 12927 rpcz_store.cc:276] 0620 23:00:40.159200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:40.159274 (+ 74us) service_pool.cc:227] Handling call
0620 23:00:41.979915 (+1820641us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:40.581355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:40.581527 (+ 172us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:40.581534 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:40.581536 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:40.581631 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:40.581642 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:40.581647 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:40.581728 (+ 81us) tablet.cc:699] Row locks acquired
0620 23:00:40.581731 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:40.581779 (+ 48us) write_op.cc:270] Start()
0620 23:00:41.547940 (+966161us) write_op.cc:276] Timestamp: P: 1781996440581778 usec, L: 0
0620 23:00:41.547946 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:41.548175 (+ 229us) log.cc:842] Serialized 354 byte log entry
0620 23:00:41.978198 (+430023us) op_driver.cc:444] REPLICATION: finished
0620 23:00:41.979216 (+ 1018us) write_op.cc:301] APPLY: starting
0620 23:00:41.979233 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:41.979277 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:41.979317 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:41.979435 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:41.979543 (+ 108us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:41.979550 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:00:41.979688 (+ 138us) log.cc:842] Serialized 22 byte log entry
0620 23:00:41.979798 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:41.979815 (+ 17us) write_op.cc:454] Released schema lock
0620 23:00:41.979822 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":958,"num_ops":1,"prepare.queue_time_us":421698,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":966884,"replication_time_us":430224,"thread_start_us":573,"threads_started":1}]]}
I20260620 23:00:42.258728 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:42.258942 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:42.259634 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:42.260241 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:42.260339 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:42.261031 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:42.543049 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:42.545378 12931 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=26, attempt_no=0}) took 1425 ms. Trace:
I20260620 23:00:42.545660 12931 rpcz_store.cc:276] 0620 23:00:41.119785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:41.119858 (+ 73us) service_pool.cc:227] Handling call
0620 23:00:42.545316 (+1425458us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:41.548268 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:41.548412 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:41.548420 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:00:41.548424 (+ 4us) tablet.cc:660] Decoding operations
0620 23:00:41.548527 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:41.548540 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:41.548545 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:41.548602 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:00:41.548604 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:41.548648 (+ 44us) write_op.cc:270] Start()
0620 23:00:41.670803 (+122155us) write_op.cc:276] Timestamp: P: 1781996441548838 usec, L: 0
0620 23:00:41.670808 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:41.671031 (+ 223us) log.cc:842] Serialized 354 byte log entry
0620 23:00:42.544197 (+873166us) op_driver.cc:444] REPLICATION: finished
0620 23:00:42.544727 (+ 530us) write_op.cc:301] APPLY: starting
0620 23:00:42.544746 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:42.544802 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:42.544844 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:42.544937 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:42.544983 (+ 46us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:42.544990 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:00:42.545090 (+ 100us) log.cc:842] Serialized 22 byte log entry
0620 23:00:42.545178 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:42.545200 (+ 22us) write_op.cc:454] Released schema lock
0620 23:00:42.545208 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":460,"num_ops":1,"prepare.queue_time_us":427933,"prepare.run_cpu_time_us":765,"prepare.run_wall_time_us":122831,"replication_time_us":873357,"thread_start_us":381,"threads_started":1}]]}
I20260620 23:00:42.569671 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:42.767007 12806 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:42.767200 12806 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:42.767967 12806 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:42.768525 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:42.768572 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:42.769233 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:42.769490 12806 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:43.159523 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:43.159718 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:43.160420 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:43.160802 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:43.161087 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:43.161608 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:43.161926 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:43.588542 12670 tablet_service.cc:2970] error setting up scanner 9b4e79e1e07d4ad7b676cad2210d9523 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.108s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:43.852427 12937 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:43.852684 12937 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:43.853647 12937 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:43.854261 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:43.854275 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:43.854982 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:43.855094 12940 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=28, attempt_no=0}) took 1306 ms. Trace:
I20260620 23:00:43.855352 12937 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:43.855243 12940 rpcz_store.cc:276] 0620 23:00:42.548588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:42.548641 (+ 53us) service_pool.cc:227] Handling call
0620 23:00:43.855043 (+1306402us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:42.622102 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:42.622234 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:42.622240 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:42.622243 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:42.622363 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:42.622375 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:42.622380 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:42.622431 (+ 51us) tablet.cc:699] Row locks acquired
0620 23:00:42.622433 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:42.622487 (+ 54us) write_op.cc:270] Start()
0620 23:00:43.479639 (+857152us) write_op.cc:276] Timestamp: P: 1781996442622473 usec, L: 0
0620 23:00:43.479645 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:43.479967 (+ 322us) log.cc:842] Serialized 354 byte log entry
0620 23:00:43.853639 (+373672us) op_driver.cc:444] REPLICATION: finished
0620 23:00:43.854166 (+ 527us) write_op.cc:301] APPLY: starting
0620 23:00:43.854185 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:43.854245 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:43.854309 (+ 64us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:43.854433 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:43.854468 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:43.854474 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:43.854638 (+ 164us) log.cc:842] Serialized 22 byte log entry
0620 23:00:43.854918 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:43.854937 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:43.854944 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":469,"num_ops":1,"prepare.queue_time_us":73054,"prepare.run_cpu_time_us":920,"prepare.run_wall_time_us":857976,"raft.queue_time_us":44,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":233,"replication_time_us":373965,"spinlock_wait_cycles":161664,"thread_start_us":355,"threads_started":1}]]}
I20260620 23:00:44.446398 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:44.446590 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:44.447443 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:44.447929 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:44.447979 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:44.448661 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:44.448930 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:45.167088 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:45.167292 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:45.168040 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:45.168578 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:45.168650 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:45.169268 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:45.169510 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:45.170989 12951 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=29, attempt_no=0}) took 2400 ms. Trace:
I20260620 23:00:45.171103 12951 rpcz_store.cc:276] 0620 23:00:42.770103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:42.770305 (+ 202us) service_pool.cc:227] Handling call
0620 23:00:45.170945 (+2400640us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:43.480111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:43.480234 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:43.480240 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:43.480242 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:43.480337 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:43.480350 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:43.480354 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:43.480409 (+ 55us) tablet.cc:699] Row locks acquired
0620 23:00:43.480411 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:43.480473 (+ 62us) write_op.cc:270] Start()
0620 23:00:44.456604 (+976131us) write_op.cc:276] Timestamp: P: 1781996443480608 usec, L: 0
0620 23:00:44.456608 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:00:44.456772 (+ 164us) log.cc:842] Serialized 354 byte log entry
0620 23:00:45.169825 (+713053us) op_driver.cc:444] REPLICATION: finished
0620 23:00:45.170390 (+ 565us) write_op.cc:301] APPLY: starting
0620 23:00:45.170408 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:45.170452 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:45.170484 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:45.170612 (+ 128us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:45.170638 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:45.170643 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:00:45.170768 (+ 125us) log.cc:842] Serialized 22 byte log entry
0620 23:00:45.170829 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:45.170843 (+ 14us) write_op.cc:454] Released schema lock
0620 23:00:45.170849 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":502,"num_ops":1,"prepare.queue_time_us":709356,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":976801,"raft.queue_time_us":450,"raft.run_cpu_time_us":545,"raft.run_wall_time_us":545,"replication_time_us":713196,"thread_start_us":390,"threads_started":1}]]}
I20260620 23:00:45.434612 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:45.434837 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:45.435695 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:45.436203 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:45.436442 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:45.437003 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
W20260620 23:00:45.675069 12670 tablet_service.cc:2970] error setting up scanner 347b4826e57548a6a0f819bdacd28559 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.504s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:45.814118 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:45.814329 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:45.815085 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:45.815680 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:45.815716 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:45.816421 12957 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=30, attempt_no=0}) took 1959 ms. Trace:
I20260620 23:00:45.816684 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:45.816599 12957 rpcz_store.cc:276] 0620 23:00:43.857330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:43.857391 (+ 61us) service_pool.cc:227] Handling call
0620 23:00:45.816360 (+1958969us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:44.456949 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:44.457079 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:44.457084 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:44.457086 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:44.457179 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:44.457190 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:44.457195 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:44.457268 (+ 73us) tablet.cc:699] Row locks acquired
0620 23:00:44.457271 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:44.457319 (+ 48us) write_op.cc:270] Start()
0620 23:00:44.970465 (+513146us) write_op.cc:276] Timestamp: P: 1781996444457437 usec, L: 0
0620 23:00:44.970470 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:44.970733 (+ 263us) log.cc:842] Serialized 354 byte log entry
0620 23:00:45.815180 (+844447us) op_driver.cc:444] REPLICATION: finished
0620 23:00:45.815864 (+ 684us) write_op.cc:301] APPLY: starting
0620 23:00:45.815882 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:45.815926 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:45.815951 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:45.816027 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:45.816059 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:45.816065 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:45.816146 (+ 81us) log.cc:842] Serialized 22 byte log entry
0620 23:00:45.816222 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:45.816251 (+ 29us) write_op.cc:454] Released schema lock
0620 23:00:45.816258 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":605,"num_ops":1,"prepare.queue_time_us":599100,"prepare.run_cpu_time_us":830,"prepare.run_wall_time_us":513890,"replication_time_us":844670,"thread_start_us":933,"threads_started":1}]]}
I20260620 23:00:45.885799 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:46.349121 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:46.349331 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:46.350029 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:46.350369 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:46.350606 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:46.351152 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:46.351481 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:46.576560 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:46.577194 12450 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 3203) took 1140 ms. Trace:
I20260620 23:00:46.577330 12450 rpcz_store.cc:276] 0620 23:00:45.436615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:45.436926 (+ 311us) service_pool.cc:227] Handling call
0620 23:00:45.437145 (+ 219us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:45.437283 (+ 138us) raft_consensus.cc:1520] Early marking committed up to index 31
0620 23:00:45.437634 (+ 351us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:45.437750 (+ 116us) write_op.cc:270] Start()
0620 23:00:45.700912 (+263162us) write_op.cc:276] Timestamp: P: 1781996444457437 usec, L: 0
0620 23:00:45.701223 (+ 311us) write_op.cc:270] Start()
0620 23:00:46.575376 (+874153us) write_op.cc:276] Timestamp: P: 1781996445173536 usec, L: 0
0620 23:00:46.576047 (+ 671us) log.cc:842] Serialized 708 byte log entry
0620 23:00:46.576432 (+ 385us) raft_consensus.cc:1631] Marking committed up to 31
0620 23:00:46.576440 (+ 8us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:46.576477 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:46.577116 (+ 639us) raft_consensus.cc:1671] finished
0620 23:00:46.577118 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:46.577160 (+ 42us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:45.701163 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:45.701272 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:45.701277 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:45.701279 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:45.701363 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:45.701377 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:45.701380 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:45.701448 (+ 68us) tablet.cc:699] Row locks acquired
0620 23:00:45.701450 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:00:46.575940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:46.576069 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:46.576076 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:46.576078 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:46.576185 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:46.576208 (+ 23us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:46.576216 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:46.576344 (+ 128us) tablet.cc:699] Row locks acquired
0620 23:00:46.576349 (+ 5us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":301,"threads_started":1,"wal-append.queue_time_us":472,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":397,"prepare.run_wall_time_us":402}],["op",{"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":438,"prepare.run_cpu_time_us":531,"prepare.run_wall_time_us":530,"thread_start_us":249,"threads_started":1}]]}
I20260620 23:00:47.231446 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:47.231662 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:47.232424 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:47.232923 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:47.233199 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:47.233786 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:47.234194 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:47.235522 12967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=32, attempt_no=0}) took 1417 ms. Trace:
I20260620 23:00:47.235737 12967 rpcz_store.cc:276] 0620 23:00:45.818489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:45.818553 (+ 64us) service_pool.cc:227] Handling call
0620 23:00:47.235472 (+1416919us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:45.819025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:45.819113 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:45.819117 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:00:45.819118 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:45.819183 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:45.819191 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:45.819194 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:45.819233 (+ 39us) tablet.cc:699] Row locks acquired
0620 23:00:45.819235 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:45.819268 (+ 33us) write_op.cc:270] Start()
0620 23:00:46.717397 (+898129us) write_op.cc:276] Timestamp: P: 1781996445819320 usec, L: 0
0620 23:00:46.717403 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:46.717643 (+ 240us) log.cc:842] Serialized 354 byte log entry
0620 23:00:47.234110 (+516467us) op_driver.cc:444] REPLICATION: finished
0620 23:00:47.234807 (+ 697us) write_op.cc:301] APPLY: starting
0620 23:00:47.234843 (+ 36us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:47.234894 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:47.234930 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:47.235042 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:47.235085 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:47.235091 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:47.235210 (+ 119us) log.cc:842] Serialized 22 byte log entry
0620 23:00:47.235325 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:47.235344 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:47.235352 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":446,"num_ops":1,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":673,"prepare.run_wall_time_us":898729,"raft.queue_time_us":70,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":236,"replication_time_us":516680,"thread_start_us":335,"threads_started":1}]]}
I20260620 23:00:47.310098 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:47.310328 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:47.311111 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:47.311661 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:47.311990 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:47.312500 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:47.312776 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:47.431550 12670 tablet_service.cc:2970] error setting up scanner ee8e0b88a3a94bc0807b0be24348daf4 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.120s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:47.732213 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:47.732434 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:47.733237 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:47.733611 12967 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=33, attempt_no=0}) took 1844 ms. Trace:
I20260620 23:00:47.733850 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:47.733866 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:47.733747 12967 rpcz_store.cc:276] 0620 23:00:45.889461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:45.889537 (+ 76us) service_pool.cc:227] Handling call
0620 23:00:47.733526 (+1843989us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:46.717815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:46.717963 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:46.717969 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:46.717971 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:46.718073 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:46.718085 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:46.718090 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:46.718149 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:00:46.718152 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:46.718205 (+ 53us) write_op.cc:270] Start()
0620 23:00:47.578371 (+860166us) write_op.cc:276] Timestamp: P: 1781996446718201 usec, L: 0
0620 23:00:47.578383 (+ 12us) op_driver.cc:331] REPLICATION: starting
0620 23:00:47.578607 (+ 224us) log.cc:842] Serialized 354 byte log entry
0620 23:00:47.732882 (+154275us) op_driver.cc:444] REPLICATION: finished
0620 23:00:47.732970 (+ 88us) write_op.cc:301] APPLY: starting
0620 23:00:47.732989 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:47.733035 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:47.733077 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:47.733187 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:47.733224 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:47.733231 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:00:47.733317 (+ 86us) log.cc:842] Serialized 22 byte log entry
0620 23:00:47.733394 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:47.733413 (+ 19us) write_op.cc:454] Released schema lock
0620 23:00:47.733421 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"num_ops":1,"prepare.queue_time_us":827796,"prepare.run_cpu_time_us":804,"prepare.run_wall_time_us":860871,"replication_time_us":154468}]]}
I20260620 23:00:47.734711 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:47.734957 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:48.558418 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:48.558648 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:48.559341 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:48.560021 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:48.560056 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:48.560652 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:48.681793 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:48.682027 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:48.682829 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:48.683420 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:48.683374 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:48.684113 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:48.684523 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:48.685156 12979 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=34, attempt_no=0}) took 1447 ms. Trace:
I20260620 23:00:48.685310 12979 rpcz_store.cc:276] 0620 23:00:47.237673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:47.237737 (+ 64us) service_pool.cc:227] Handling call
0620 23:00:48.685090 (+1447353us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:47.578766 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:47.578894 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:47.578900 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:47.578904 (+ 4us) tablet.cc:660] Decoding operations
0620 23:00:47.578994 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:47.579004 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:47.579008 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:47.579067 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:00:47.579069 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:47.579121 (+ 52us) write_op.cc:270] Start()
0620 23:00:47.818275 (+239154us) write_op.cc:276] Timestamp: P: 1781996447579254 usec, L: 0
0620 23:00:47.818279 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:00:47.818486 (+ 207us) log.cc:842] Serialized 354 byte log entry
0620 23:00:48.683092 (+864606us) op_driver.cc:444] REPLICATION: finished
0620 23:00:48.683986 (+ 894us) write_op.cc:301] APPLY: starting
0620 23:00:48.684007 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:48.684051 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:48.684093 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:48.684188 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:48.684221 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:48.684227 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:48.684324 (+ 97us) log.cc:842] Serialized 22 byte log entry
0620 23:00:48.684966 (+ 642us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:48.684988 (+ 22us) write_op.cc:454] Released schema lock
0620 23:00:48.684996 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":828,"num_ops":1,"prepare.queue_time_us":340502,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":239824,"replication_time_us":864786,"thread_start_us":1230,"threads_started":1}]]}
I20260620 23:00:49.050127 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:49.050331 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:49.051103 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:49.051641 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:49.051658 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:49.051954 12979 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=35, attempt_no=0}) took 1315 ms. Trace:
I20260620 23:00:49.052073 12979 rpcz_store.cc:276] 0620 23:00:47.736165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:47.736231 (+ 66us) service_pool.cc:227] Handling call
0620 23:00:49.051895 (+1315664us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:47.818644 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:47.818765 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:47.818772 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:47.818774 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:47.818868 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:47.818882 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:47.818887 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:47.818941 (+ 54us) tablet.cc:699] Row locks acquired
0620 23:00:47.818943 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:47.818981 (+ 38us) write_op.cc:270] Start()
0620 23:00:48.722153 (+903172us) write_op.cc:276] Timestamp: P: 1781996447819010 usec, L: 0
0620 23:00:48.722159 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:48.722352 (+ 193us) log.cc:842] Serialized 354 byte log entry
0620 23:00:49.051169 (+328817us) op_driver.cc:444] REPLICATION: finished
0620 23:00:49.051369 (+ 200us) write_op.cc:301] APPLY: starting
0620 23:00:49.051386 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:49.051422 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:49.051469 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:49.051564 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:49.051602 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:49.051613 (+ 11us) write_op.cc:312] APPLY: finished
0620 23:00:49.051712 (+ 99us) log.cc:842] Serialized 22 byte log entry
0620 23:00:49.051790 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:49.051808 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:49.051815 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":136,"num_ops":1,"prepare.queue_time_us":81979,"prepare.run_cpu_time_us":720,"prepare.run_wall_time_us":903813,"replication_time_us":328970}]]}
I20260620 23:00:49.052452 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:49.052847 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:49.195636 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:49.831574 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:49.831794 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:49.832621 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:49.833221 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:49.833237 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:49.833864 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:49.923199 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:49.923444 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:49.924242 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:49.924897 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:49.924913 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:49.925738 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:49.926255 12987 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=36, attempt_no=0}) took 1239 ms. Trace:
I20260620 23:00:49.926298 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:49.926407 12987 rpcz_store.cc:276] 0620 23:00:48.687099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:48.687166 (+ 67us) service_pool.cc:227] Handling call
0620 23:00:49.926196 (+1239030us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:48.722515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:48.722644 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:48.722650 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:48.722653 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:48.722760 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:48.722773 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:48.722777 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:48.722836 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:00:48.722839 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:48.722892 (+ 53us) write_op.cc:270] Start()
0620 23:00:49.510055 (+787163us) write_op.cc:276] Timestamp: P: 1781996448722872 usec, L: 0
0620 23:00:49.510060 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:49.510307 (+ 247us) log.cc:842] Serialized 354 byte log entry
0620 23:00:49.924550 (+414243us) op_driver.cc:444] REPLICATION: finished
0620 23:00:49.925608 (+ 1058us) write_op.cc:301] APPLY: starting
0620 23:00:49.925632 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:49.925688 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:49.925724 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:49.925799 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:49.925834 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:49.925840 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:49.925960 (+ 120us) log.cc:842] Serialized 22 byte log entry
0620 23:00:49.926046 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:49.926067 (+ 21us) write_op.cc:454] Released schema lock
0620 23:00:49.926075 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":982,"num_ops":1,"prepare.queue_time_us":34856,"prepare.run_cpu_time_us":812,"prepare.run_wall_time_us":787892,"replication_time_us":414462,"thread_start_us":863,"threads_started":1}]]}
W20260620 23:00:50.250618 12670 tablet_service.cc:2970] error setting up scanner 490361a0b1024a0dbd59d3738e054079 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.325s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
W20260620 23:00:50.286254 12670 tablet_service.cc:2970] error setting up scanner f2057aeef40640e6bc68fd4c29e5c8e9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.361s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:50.383857 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:50.384089 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:50.384886 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:50.385473 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:50.385525 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:50.385685 12987 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=37, attempt_no=0}) took 1331 ms. Trace:
I20260620 23:00:50.385788 12987 rpcz_store.cc:276] 0620 23:00:49.054600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:49.054662 (+ 62us) service_pool.cc:227] Handling call
0620 23:00:50.385618 (+1330956us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:49.510458 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:49.510586 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:49.510593 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:49.510595 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:49.510687 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:49.510702 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:49.510707 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:49.510763 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:00:49.510764 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:49.510804 (+ 40us) write_op.cc:270] Start()
0620 23:00:50.111970 (+601166us) write_op.cc:276] Timestamp: P: 1781996449510896 usec, L: 0
0620 23:00:50.111976 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:50.112225 (+ 249us) log.cc:842] Serialized 354 byte log entry
0620 23:00:50.384889 (+272664us) op_driver.cc:444] REPLICATION: finished
0620 23:00:50.385035 (+ 146us) write_op.cc:301] APPLY: starting
0620 23:00:50.385057 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:50.385103 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:50.385136 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:50.385220 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:50.385256 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:50.385262 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:50.385357 (+ 95us) log.cc:842] Serialized 22 byte log entry
0620 23:00:50.385474 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:50.385495 (+ 21us) write_op.cc:454] Released schema lock
0620 23:00:50.385505 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63,"num_ops":1,"prepare.queue_time_us":455359,"prepare.run_cpu_time_us":790,"prepare.run_wall_time_us":601845,"replication_time_us":272882}]]}
I20260620 23:00:50.386222 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:50.386590 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:50.819826 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:51.259052 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:51.259256 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:51.260103 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:51.260717 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:51.260815 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:51.261690 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:51.262014 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:51.262998 12993 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=38, attempt_no=0}) took 1334 ms. Trace:
I20260620 23:00:51.263137 12993 rpcz_store.cc:276] 0620 23:00:49.928369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:49.928440 (+ 71us) service_pool.cc:227] Handling call
0620 23:00:51.262946 (+1334506us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:50.112354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:50.112478 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:50.112483 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:50.112484 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:50.112587 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:50.112598 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:50.112604 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:50.112672 (+ 68us) tablet.cc:699] Row locks acquired
0620 23:00:50.112674 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:50.112725 (+ 51us) write_op.cc:270] Start()
0620 23:00:50.501880 (+389155us) write_op.cc:276] Timestamp: P: 1781996450112852 usec, L: 0
0620 23:00:50.501884 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:00:50.502162 (+ 278us) log.cc:842] Serialized 354 byte log entry
0620 23:00:51.260520 (+758358us) op_driver.cc:444] REPLICATION: finished
0620 23:00:51.262386 (+ 1866us) write_op.cc:301] APPLY: starting
0620 23:00:51.262402 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:51.262437 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:51.262499 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:51.262582 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:51.262605 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:51.262610 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:00:51.262707 (+ 97us) log.cc:842] Serialized 22 byte log entry
0620 23:00:51.262826 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:51.262843 (+ 17us) write_op.cc:454] Released schema lock
0620 23:00:51.262850 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1805,"num_ops":1,"prepare.queue_time_us":183537,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":389913,"replication_time_us":758609,"thread_start_us":1043,"threads_started":1}]]}
I20260620 23:00:52.322860 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:52.323092 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:52.323767 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:52.324270 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:52.324579 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:52.325150 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:52.325611 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:52.326261 13000 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=39, attempt_no=0}) took 1938 ms. Trace:
I20260620 23:00:52.326423 13000 rpcz_store.cc:276] 0620 23:00:50.387845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:50.387910 (+ 65us) service_pool.cc:227] Handling call
0620 23:00:52.326207 (+1938297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:50.502306 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:50.502441 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:50.502447 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:50.502450 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:50.502574 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:50.502587 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:50.502593 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:50.502656 (+ 63us) tablet.cc:699] Row locks acquired
0620 23:00:50.502658 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:50.502697 (+ 39us) write_op.cc:270] Start()
0620 23:00:51.437873 (+935176us) write_op.cc:276] Timestamp: P: 1781996450502784 usec, L: 0
0620 23:00:51.437878 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:51.438087 (+ 209us) log.cc:842] Serialized 354 byte log entry
0620 23:00:52.324915 (+886828us) op_driver.cc:444] REPLICATION: finished
0620 23:00:52.325568 (+ 653us) write_op.cc:301] APPLY: starting
0620 23:00:52.325588 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:52.325632 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:52.325662 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:52.325787 (+ 125us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:52.325831 (+ 44us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:52.325837 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:52.325958 (+ 121us) log.cc:842] Serialized 22 byte log entry
0620 23:00:52.326040 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:52.326058 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:52.326065 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":590,"num_ops":1,"prepare.queue_time_us":113842,"prepare.run_cpu_time_us":817,"prepare.run_wall_time_us":935869,"replication_time_us":887011,"thread_start_us":324,"threads_started":1}]]}
I20260620 23:00:52.430234 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:52.430424 12449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 3535) took 1608 ms. Trace:
I20260620 23:00:52.430470 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:52.431305 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:52.430629 12449 rpcz_store.cc:276] 0620 23:00:50.821641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:50.821727 (+ 86us) service_pool.cc:227] Handling call
0620 23:00:50.822039 (+ 312us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:50.822151 (+ 112us) raft_consensus.cc:1520] Early marking committed up to index 38
0620 23:00:50.822551 (+ 400us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:50.822691 (+ 140us) write_op.cc:270] Start()
0620 23:00:51.765860 (+943169us) write_op.cc:276] Timestamp: P: 1781996449510896 usec, L: 0
0620 23:00:51.766429 (+ 569us) write_op.cc:270] Start()
0620 23:00:52.428636 (+662207us) write_op.cc:276] Timestamp: P: 1781996450112852 usec, L: 0
0620 23:00:52.429340 (+ 704us) log.cc:842] Serialized 708 byte log entry
0620 23:00:52.429723 (+ 383us) raft_consensus.cc:1631] Marking committed up to 39
0620 23:00:52.430098 (+ 375us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:52.430135 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:52.430307 (+ 172us) raft_consensus.cc:1671] finished
0620 23:00:52.430310 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:52.430369 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:51.766396 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:51.766561 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:51.766566 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:51.766568 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:51.766693 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:51.766713 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:51.766719 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:51.766782 (+ 63us) tablet.cc:699] Row locks acquired
0620 23:00:51.766784 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:52.429766 (+662982us) op_driver.cc:444] REPLICATION: finished
0620 23:00:52.430216 (+ 450us) write_op.cc:301] APPLY: starting
0620 23:00:52.430239 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:52.430296 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:52.430298 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:52.430366 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:52.430493 (+ 127us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:52.430535 (+ 42us) write_op.cc:312] APPLY: finished
0620 23:00:52.430629 (+ 94us) log.cc:842] Serialized 22 byte log entry
0620 23:00:52.430691 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:52.430706 (+ 15us) write_op.cc:454] Released schema lock
0620 23:00:52.430712 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0620 23:00:52.429233 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:52.429393 (+ 160us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:52.429399 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:52.429401 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:52.429493 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:52.429506 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:52.429509 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:52.429562 (+ 53us) tablet.cc:699] Row locks acquired
0620 23:00:52.429564 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":296,"threads_started":1,"wal-append.queue_time_us":382,"child_traces":[["op",{"apply.queue_time_us":385,"apply.run_cpu_time_us":574,"apply.run_wall_time_us":641,"num_ops":1,"partition_lock_wait_us":10,"prepare.queue_time_us":334,"prepare.run_cpu_time_us":485,"prepare.run_wall_time_us":485,"replication_time_us":1607060,"thread_start_us":477,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":460,"prepare.run_cpu_time_us":419,"prepare.run_wall_time_us":419,"thread_start_us":317,"threads_started":1}]]}
I20260620 23:00:52.431965 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:52.431993 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:52.432690 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:52.433003 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:52.713225 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:52.713435 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:52.714277 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:52.714819 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:52.714843 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:52.715515 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:52.715724 13000 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=40, attempt_no=0}) took 1450 ms. Trace:
I20260620 23:00:52.715935 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:52.715834 13000 rpcz_store.cc:276] 0620 23:00:51.265116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:51.265172 (+ 56us) service_pool.cc:227] Handling call
0620 23:00:52.715653 (+1450481us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:51.438216 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:51.438347 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:51.438355 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:00:51.438357 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:51.438479 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:51.438492 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:51.438500 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:51.438606 (+ 106us) tablet.cc:699] Row locks acquired
0620 23:00:51.438611 (+ 5us) write_op.cc:260] PREPARE: finished
0620 23:00:51.438662 (+ 51us) write_op.cc:270] Start()
0620 23:00:51.732828 (+294166us) write_op.cc:276] Timestamp: P: 1781996451438698 usec, L: 0
0620 23:00:51.732833 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:51.733081 (+ 248us) log.cc:842] Serialized 354 byte log entry
0620 23:00:52.714381 (+981300us) op_driver.cc:444] REPLICATION: finished
0620 23:00:52.714506 (+ 125us) write_op.cc:301] APPLY: starting
0620 23:00:52.714525 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:52.714569 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:52.714602 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:52.714681 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:52.714720 (+ 39us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:52.714728 (+ 8us) write_op.cc:312] APPLY: finished
0620 23:00:52.715249 (+ 521us) log.cc:842] Serialized 22 byte log entry
0620 23:00:52.715533 (+ 284us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:52.715551 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:52.715558 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"num_ops":1,"prepare.queue_time_us":172680,"prepare.run_cpu_time_us":922,"prepare.run_wall_time_us":294991,"replication_time_us":981509}]]}
I20260620 23:00:53.057008 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:53.057224 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:53.058022 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:53.058648 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:53.058648 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:53.059370 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:53.453096 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:53.492681 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:53.492910 12449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 3536) took 1060 ms. Trace:
I20260620 23:00:53.492882 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:53.493682 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:53.493096 12449 rpcz_store.cc:276] 0620 23:00:52.432620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:52.432692 (+ 72us) service_pool.cc:227] Handling call
0620 23:00:52.432975 (+ 283us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:52.433273 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 40
0620 23:00:52.433342 (+ 69us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:52.433428 (+ 86us) write_op.cc:270] Start()
0620 23:00:53.249587 (+816159us) write_op.cc:276] Timestamp: P: 1781996450502784 usec, L: 0
0620 23:00:53.250186 (+ 599us) write_op.cc:270] Start()
0620 23:00:53.491390 (+241204us) write_op.cc:276] Timestamp: P: 1781996451438698 usec, L: 0
0620 23:00:53.491811 (+ 421us) log.cc:842] Serialized 708 byte log entry
0620 23:00:53.492175 (+ 364us) raft_consensus.cc:1631] Marking committed up to 41
0620 23:00:53.492534 (+ 359us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:53.492581 (+ 47us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:53.492794 (+ 213us) raft_consensus.cc:1671] finished
0620 23:00:53.492797 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:53.492854 (+ 57us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:53.250132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:53.250248 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:53.250253 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:00:53.250255 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:53.250341 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:53.250354 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:53.250357 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:53.250404 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:00:53.250405 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:00:53.492215 (+241810us) op_driver.cc:444] REPLICATION: finished
0620 23:00:53.492683 (+ 468us) write_op.cc:301] APPLY: starting
0620 23:00:53.492700 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:53.492743 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:53.492746 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:53.492815 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:53.492847 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:53.492898 (+ 51us) write_op.cc:312] APPLY: finished
0620 23:00:53.492979 (+ 81us) log.cc:842] Serialized 22 byte log entry
0620 23:00:53.493066 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:53.493087 (+ 21us) write_op.cc:454] Released schema lock
0620 23:00:53.493093 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0620 23:00:53.493165 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:53.493169 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:00:53.491619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:53.491735 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:53.491741 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:53.491743 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:53.491823 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:53.491834 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:53.491837 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:53.491883 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:00:53.491885 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":393984,"thread_start_us":283,"threads_started":1,"wal-append.queue_time_us":372,"child_traces":[["op",{"apply.queue_time_us":398,"apply.run_cpu_time_us":556,"apply.run_wall_time_us":555,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":404,"prepare.run_cpu_time_us":361,"prepare.run_wall_time_us":362,"replication_time_us":1058770,"thread_start_us":517,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":355,"prepare.run_wall_time_us":356}]]}
I20260620 23:00:53.494180 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:53.494347 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:53.494999 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:53.495286 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:54.105670 12670 tablet_service.cc:2970] error setting up scanner 9dbe31cb0460420aa3d87634d76d9b07 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1781996453105468 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1781996452760513 usec, L: 0 Physical time difference: 0.345s: new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "7db9a4eefb6c4e8bb800e00c6008b044"
I20260620 23:00:54.106143 12670 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49796 (request call id 29) took 1002 ms. Trace:
I20260620 23:00:54.106264 12670 rpcz_store.cc:276] 0620 23:00:53.104056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:53.104145 (+ 89us) service_pool.cc:227] Handling call
0620 23:00:53.104386 (+ 241us) tablet_service.cc:2780] Created scanner 9dbe31cb0460420aa3d87634d76d9b07 for tablet c4447a4bda0d41f09b88af81b2668b51, query id is 7db9a4eefb6c4e8bb800e00c6008b044
0620 23:00:53.105318 (+ 932us) tablet_service.cc:2920] Creating iterator
0620 23:00:53.105348 (+ 30us) tablet_service.cc:3297] Waiting safe time to advance
0620 23:00:54.105596 (+1000248us) tablet_service.cc:2945] Iterator created
0620 23:00:54.105611 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1781996453105468 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1781996452760513 usec, L: 0 Physical time difference: 0.345s
0620 23:00:54.106103 (+ 492us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260620 23:00:54.285570 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:54.285778 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:54.286432 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:54.287079 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:54.287225 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:54.287955 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:54.288552 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:54.289870 13013 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=43, attempt_no=0}) took 1228 ms. Trace:
I20260620 23:00:54.290074 13013 rpcz_store.cc:276] 0620 23:00:53.060924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:53.060986 (+ 62us) service_pool.cc:227] Handling call
0620 23:00:54.289773 (+1228787us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:53.061541 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:53.061686 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:53.061689 (+ 3us) write_op.cc:435] Acquired schema lock
0620 23:00:53.061692 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:53.061796 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:53.061806 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:53.061817 (+ 11us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:53.061855 (+ 38us) tablet.cc:699] Row locks acquired
0620 23:00:53.061857 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:53.061888 (+ 31us) write_op.cc:270] Start()
0620 23:00:53.466002 (+404114us) write_op.cc:276] Timestamp: P: 1781996453061958 usec, L: 0
0620 23:00:53.466006 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:00:53.466224 (+ 218us) log.cc:842] Serialized 354 byte log entry
0620 23:00:54.288021 (+821797us) op_driver.cc:444] REPLICATION: finished
0620 23:00:54.289118 (+ 1097us) write_op.cc:301] APPLY: starting
0620 23:00:54.289145 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:54.289259 (+ 114us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:54.289324 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:54.289393 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:54.289419 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:54.289423 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:00:54.289526 (+ 103us) log.cc:842] Serialized 22 byte log entry
0620 23:00:54.289657 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:54.289675 (+ 18us) write_op.cc:454] Released schema lock
0620 23:00:54.289682 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1027,"num_ops":1,"prepare.queue_time_us":35,"prepare.run_cpu_time_us":719,"prepare.run_wall_time_us":404773,"replication_time_us":821988,"thread_start_us":896,"threads_started":1}]]}
I20260620 23:00:54.837951 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:54.838006 12449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 3537) took 1342 ms. Trace:
I20260620 23:00:54.838200 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:54.839165 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:54.838203 12449 rpcz_store.cc:276] 0620 23:00:53.495194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:53.495245 (+ 51us) service_pool.cc:227] Handling call
0620 23:00:53.495502 (+ 257us) raft_consensus.cc:1457] Updating replica for 3 ops
0620 23:00:53.495628 (+ 126us) raft_consensus.cc:1520] Early marking committed up to index 42
0620 23:00:53.495702 (+ 74us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0620 23:00:53.495790 (+ 88us) write_op.cc:270] Start()
0620 23:00:53.961958 (+466168us) write_op.cc:276] Timestamp: P: 1781996452329376 usec, L: 0
0620 23:00:53.962316 (+ 358us) write_op.cc:270] Start()
0620 23:00:54.408482 (+446166us) write_op.cc:276] Timestamp: P: 1781996452760513 usec, L: 0
0620 23:00:54.408829 (+ 347us) write_op.cc:270] Start()
0620 23:00:54.836011 (+427182us) write_op.cc:276] Timestamp: P: 1781996453061958 usec, L: 0
0620 23:00:54.836693 (+ 682us) log.cc:842] Serialized 1062 byte log entry
0620 23:00:54.837067 (+ 374us) raft_consensus.cc:1631] Marking committed up to 44
0620 23:00:54.837781 (+ 714us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:54.837824 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:54.837836 (+ 12us) raft_consensus.cc:1671] finished
0620 23:00:54.837839 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:54.837951 (+ 112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:53.962167 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:53.962319 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:53.962325 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:53.962327 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:53.962420 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:53.962434 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:53.962437 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:53.962507 (+ 70us) tablet.cc:699] Row locks acquired
0620 23:00:53.962509 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:54.837100 (+874591us) op_driver.cc:444] REPLICATION: finished
0620 23:00:54.837574 (+ 474us) write_op.cc:301] APPLY: starting
0620 23:00:54.837593 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:54.837649 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:54.837654 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:54.837752 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:54.837788 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:54.837862 (+ 74us) write_op.cc:312] APPLY: finished
0620 23:00:54.838042 (+ 180us) log.cc:842] Serialized 22 byte log entry
0620 23:00:54.838134 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:54.838151 (+ 17us) write_op.cc:454] Released schema lock
0620 23:00:54.838160 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0620 23:00:54.838253 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:54.838257 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:00:54.408694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:54.408834 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:54.408840 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:54.408843 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:54.408952 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:54.408964 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:54.408969 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:54.409016 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:00:54.409018 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:54.837468 (+428450us) op_driver.cc:444] REPLICATION: finished
0620 23:00:54.837872 (+ 404us) write_op.cc:301] APPLY: starting
0620 23:00:54.837889 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:54.837931 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:54.837934 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:54.838065 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:54.838277 (+ 212us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:54.838347 (+ 70us) write_op.cc:312] APPLY: finished
0620 23:00:54.838464 (+ 117us) log.cc:842] Serialized 22 byte log entry
Related trace 'op':
0620 23:00:54.836448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:54.836606 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:54.836613 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:00:54.836615 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:54.836712 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:54.836728 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:54.836732 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:54.836789 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:00:54.836791 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":288,"threads_started":1,"wal-append.queue_time_us":373,"child_traces":[["op",{"apply.queue_time_us":426,"apply.run_cpu_time_us":767,"apply.run_wall_time_us":767,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":452,"prepare.run_wall_time_us":453,"replication_time_us":1341298,"thread_start_us":286,"threads_started":1}],["op",{"apply.queue_time_us":329,"apply.run_cpu_time_us":718,"apply.run_wall_time_us":1044,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":70,"prepare.run_cpu_time_us":410,"prepare.run_wall_time_us":410,"replication_time_us":875139,"thread_start_us":205,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":288,"prepare.run_cpu_time_us":428,"prepare.run_wall_time_us":428}]]}
I20260620 23:00:54.839649 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:54.839751 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:54.840415 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:54.840667 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:00:54.861546 12670 tablet_service.cc:2970] error setting up scanner cbd63f1b71f54de99f24ee4b0c9b94a5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.572s (maximum allowed 0.100s): new_scan_request { tablet_id: "c4447a4bda0d41f09b88af81b2668b51" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\264\271\334\321\006\"3\n\005slave\022*\n 5f35bf577cb14bce9aa74c0aa35a2da3\020\001\030\001 \001(\001" signature: "<redacted>""\317\235\023\254\233T\300\335\022\300D\235\270\263W\306Y\355\tu4\260\020!\357\273(\025\310\2148i\317\024u:S\345\004g\000\324\245[\204I\360u69\240\266\233@\3075\223*_\302\242\001F*" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "738be9712cae464ca351e1f018b421d0"
I20260620 23:00:55.362392 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:55.362612 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:55.363642 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:55.364280 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:55.364275 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:55.365056 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:55.365372 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:55.366034 13022 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=44, attempt_no=0}) took 1909 ms. Trace:
I20260620 23:00:55.366163 13022 rpcz_store.cc:276] 0620 23:00:53.456233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:53.456301 (+ 68us) service_pool.cc:227] Handling call
0620 23:00:55.365974 (+1909673us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:53.466340 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:53.466463 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:53.466485 (+ 22us) write_op.cc:435] Acquired schema lock
0620 23:00:53.466487 (+ 2us) tablet.cc:660] Decoding operations
0620 23:00:53.466577 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:53.466587 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:53.466592 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:53.466645 (+ 53us) tablet.cc:699] Row locks acquired
0620 23:00:53.466647 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:00:53.466685 (+ 38us) write_op.cc:270] Start()
0620 23:00:54.465855 (+999170us) write_op.cc:276] Timestamp: P: 1781996453466717 usec, L: 0
0620 23:00:54.465861 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:00:54.466174 (+ 313us) log.cc:842] Serialized 354 byte log entry
0620 23:00:55.364411 (+898237us) op_driver.cc:444] REPLICATION: finished
0620 23:00:55.365127 (+ 716us) write_op.cc:301] APPLY: starting
0620 23:00:55.365146 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:55.365192 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:55.365224 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:55.365316 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:55.365358 (+ 42us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:55.365367 (+ 9us) write_op.cc:312] APPLY: finished
0620 23:00:55.365471 (+ 104us) log.cc:842] Serialized 22 byte log entry
0620 23:00:55.365822 (+ 351us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:55.365842 (+ 20us) write_op.cc:454] Released schema lock
0620 23:00:55.365850 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":653,"num_ops":1,"prepare.queue_time_us":9540,"prepare.run_cpu_time_us":855,"prepare.run_wall_time_us":999909,"replication_time_us":898524,"spinlock_wait_cycles":57088,"thread_start_us":543,"threads_started":1}]]}
I20260620 23:00:55.551910 12831 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:55.552162 12831 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:55.553030 12831 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:55.553582 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:55.553623 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:55.554129 13022 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48474 (ReqId={client: ca4964c683a94ef59a9bb1516e4ab673, seq_no=45, attempt_no=0}) took 1262 ms. Trace:
I20260620 23:00:55.554473 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:55.554253 13022 rpcz_store.cc:276] 0620 23:00:54.292080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:54.292158 (+ 78us) service_pool.cc:227] Handling call
0620 23:00:55.554072 (+1261914us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:54.466296 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:54.466440 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:54.466446 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:54.466449 (+ 3us) tablet.cc:660] Decoding operations
0620 23:00:54.466552 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:54.466566 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:54.466572 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:54.466634 (+ 62us) tablet.cc:699] Row locks acquired
0620 23:00:54.466637 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:00:54.466689 (+ 52us) write_op.cc:270] Start()
0620 23:00:54.562842 (+ 96153us) write_op.cc:276] Timestamp: P: 1781996454466706 usec, L: 0
0620 23:00:54.562847 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:00:54.563120 (+ 273us) log.cc:842] Serialized 354 byte log entry
0620 23:00:55.553370 (+990250us) op_driver.cc:444] REPLICATION: finished
0620 23:00:55.553496 (+ 126us) write_op.cc:301] APPLY: starting
0620 23:00:55.553516 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:00:55.553559 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:00:55.553592 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:00:55.553684 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:00:55.553722 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:00:55.553728 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:00:55.553813 (+ 85us) log.cc:842] Serialized 22 byte log entry
0620 23:00:55.553915 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:00:55.553967 (+ 52us) write_op.cc:454] Released schema lock
0620 23:00:55.553975 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":68,"num_ops":1,"prepare.queue_time_us":173761,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":96897,"replication_time_us":990500}]]}
I20260620 23:00:55.554929 12831 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:56.091183 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.091392 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.091673 12449 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:46991 (request call id 3538) took 1251 ms. Trace:
I20260620 23:00:56.092368 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.093052 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:56.093124 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.093736 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:56.091840 12449 rpcz_store.cc:276] 0620 23:00:54.840455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:00:54.840526 (+ 71us) service_pool.cc:227] Handling call
0620 23:00:54.840754 (+ 228us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:00:54.840975 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 45
0620 23:00:54.841041 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:00:54.841121 (+ 80us) write_op.cc:270] Start()
0620 23:00:55.331293 (+490172us) write_op.cc:276] Timestamp: P: 1781996453466717 usec, L: 0
0620 23:00:55.331584 (+ 291us) write_op.cc:270] Start()
0620 23:00:56.085757 (+754173us) write_op.cc:276] Timestamp: P: 1781996454466706 usec, L: 0
0620 23:00:56.090477 (+ 4720us) log.cc:842] Serialized 708 byte log entry
0620 23:00:56.091060 (+ 583us) raft_consensus.cc:1631] Marking committed up to 45
0620 23:00:56.091069 (+ 9us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:00:56.091109 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:00:56.091567 (+ 458us) raft_consensus.cc:1671] finished
0620 23:00:56.091571 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:00:56.091620 (+ 49us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:00:55.331510 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:55.331626 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:55.331632 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:55.331633 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:55.331713 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:55.331727 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:55.331730 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:55.331777 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:00:55.331779 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:00:56.090299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c4447a4bda0d41f09b88af81b2668b51
0620 23:00:56.090409 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:00:56.090415 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:00:56.090416 (+ 1us) tablet.cc:660] Decoding operations
0620 23:00:56.090493 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:00:56.090504 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:00:56.090508 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:00:56.090586 (+ 78us) tablet.cc:699] Row locks acquired
0620 23:00:56.090589 (+ 3us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":125696,"thread_start_us":482,"threads_started":1,"wal-append.queue_time_us":585,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":375,"prepare.run_wall_time_us":376}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":4403,"prepare.run_cpu_time_us":401,"prepare.run_wall_time_us":400,"thread_start_us":4256,"threads_started":1}]]}
I20260620 23:00:56.094072 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:56.183882 13028 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.184043 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.184185 13028 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.184505 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.184890 13028 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:56.185362 12581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.185364 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:56.185580 12450 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49a8d52878a43aa83df884ff96e687e in term 1.
I20260620 23:00:56.185616 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.186168 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.186126 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd, e49a8d52878a43aa83df884ff96e687e; no voters: c707c04fc03d49da82f97968100c19d0
I20260620 23:00:56.186864 13028 raft_consensus.cc:2781] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:00:56.186861 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:56.187071 13028 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.187328 13028 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:00:56.187386 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:56.187754 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:00:56.190071 13028 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.190524 12711 raft_consensus.cc:1226] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 2 FOLLOWER]: Rejecting Update request from peer c707c04fc03d49da82f97968100c19d0 for earlier term 1. Current term is 2. Ops: []
I20260620 23:00:56.191114 13028 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 election: Requested vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:56.191063 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0"
I20260620 23:00:56.191430 13020 consensus_queue.cc:1046] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 }, Status: INVALID_TERM, Last received: 1.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260620 23:00:56.191888 13021 raft_consensus.cc:3032] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:00:56.192004 13021 raft_consensus.cc:729] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c707c04fc03d49da82f97968100c19d0, State: Running, Role: LEADER
I20260620 23:00:56.192220 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd"
I20260620 23:00:56.192348 13021 consensus_queue.cc:260] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 47, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.192658 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:56.192967 13028 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:00:56.193041 13021 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:00:56.197563 13021 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:00:56.197710 13021 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.198531 13021 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.198902 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:56.198966 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:56.199046 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.199187 12711 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 2.
I20260620 23:00:56.199179 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.199486 12450 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 1.
I20260620 23:00:56.199611 12514 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e; no voters:
I20260620 23:00:56.199841 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.200042 13021 raft_consensus.cc:2781] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:00:56.200169 13021 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:00:56.200110 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.200271 13021 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:00:56.200227 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:56.200433 12711 raft_consensus.cc:2379] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b1834d241b74f0f8152bb84b9e628dd in current term 2: Already voted for candidate e49a8d52878a43aa83df884ff96e687e in this term.
I20260620 23:00:56.202301 13021 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.202754 12580 raft_consensus.cc:2354] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b1834d241b74f0f8152bb84b9e628dd for earlier term 2. Current term is 3.
I20260620 23:00:56.202895 13021 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 3 election: Requested vote from peers 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.203395 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd"
I20260620 23:00:56.203600 12450 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 1 FOLLOWER]: Advancing to term 3
I20260620 23:00:56.203585 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "c707c04fc03d49da82f97968100c19d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e"
I20260620 23:00:56.203730 12711 raft_consensus.cc:3037] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:00:56.205668 12711 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 3.
I20260620 23:00:56.205659 12450 raft_consensus.cc:2454] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707c04fc03d49da82f97968100c19d0 in term 3.
I20260620 23:00:56.206017 12514 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e; no voters:
I20260620 23:00:56.206158 12383 leader_election.cc:398] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Vote denied by peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755) with higher term. Message: Invalid argument: T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b1834d241b74f0f8152bb84b9e628dd for earlier term 2. Current term is 3.
I20260620 23:00:56.206291 12383 leader_election.cc:401] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260620 23:00:56.206298 13021 raft_consensus.cc:2781] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:00:56.206431 13021 raft_consensus.cc:686] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 3 LEADER]: Becoming Leader. State: Replica: c707c04fc03d49da82f97968100c19d0, State: Running, Role: LEADER
I20260620 23:00:56.206519 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755) with higher term. Message: Invalid argument: T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b1834d241b74f0f8152bb84b9e628dd for earlier term 2. Current term is 3.
I20260620 23:00:56.206693 13021 consensus_queue.cc:237] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 47, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.208757 12308 catalog_manager.cc:5495] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c707c04fc03d49da82f97968100c19d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } health_report { overall_health: UNKNOWN } } }
I20260620 23:00:56.208998 12450 raft_consensus.cc:1261] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Refusing update from remote peer c707c04fc03d49da82f97968100c19d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 3 index: 48. (index mismatch)
I20260620 23:00:56.209520 13021 consensus_queue.cc:1035] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260620 23:00:56.210767 12711 raft_consensus.cc:1261] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Refusing update from remote peer c707c04fc03d49da82f97968100c19d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 3 index: 48. (index mismatch)
I20260620 23:00:56.211371 13021 consensus_queue.cc:1035] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20260620 23:00:56.219924 13028 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.220148 13028 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.220956 13028 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:56.221637 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.221843 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:56.222391 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:56.222771 13028 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:56.224673 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.224833 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.225524 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.225986 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.226702 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:56.227175 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:56.227448 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:56.236574 13028 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.236730 13028 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.237274 13028 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:56.237921 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.239167 12947 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.239343 12947 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.239912 12947 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:56.240515 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:56.240942 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:56.241389 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:56.241465 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.241653 13028 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:56.242209 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:56.242461 12947 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:00:56.933561 12803 scanner-internal.cc:458] Time spent opening tablet: real 31.890s user 0.009s sys 0.000s
I20260620 23:00:56.934295 13037 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:56.934552 13037 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:56.935986 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:56.936039 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:56.935829 13037 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:56.936910 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:56.937173 13037 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:57.216976 13037 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:57.217163 13037 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:57.217790 13037 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:57.218264 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:57.218356 12449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:57.218968 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:57.219242 13037 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:57.225919 13037 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:57.226135 13037 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:57.226770 13037 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:57.227458 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:57.227574 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:57.228200 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:57.228482 13037 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:57.233073 13038 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:57.233219 13038 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:57.234265 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:57.234541 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:57.234871 13038 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:57.235169 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:57.236099 13038 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:00:59.220876 12804 scanner-internal.cc:458] Time spent opening tablet: real 34.177s user 0.008s sys 0.003s
I20260620 23:00:59.337037 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.337325 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.339339 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:59.339571 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:59.340483 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:59.346578 12430 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20260620 23:00:59.355973 12644 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.008s (SENT) [suppressed 1 similar messages]
I20260620 23:00:59.361268 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:59.361871 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.362270 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.365796 12561 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260620 23:00:59.374578 13067 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.374811 13067 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.375732 13067 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), e49a8d52878a43aa83df884ff96e687e (127.10.134.131:44497)
I20260620 23:00:59.376228 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:59.377252 12710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "2b1834d241b74f0f8152bb84b9e628dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "e49a8d52878a43aa83df884ff96e687e" is_pre_election: true
I20260620 23:00:59.377786 12383 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1834d241b74f0f8152bb84b9e628dd; no voters: c707c04fc03d49da82f97968100c19d0, e49a8d52878a43aa83df884ff96e687e
I20260620 23:00:59.378105 13067 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.378377 12691 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:00:59.385968 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.386157 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.386821 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:00:59.392288 12580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:59.392421 12450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2b1834d241b74f0f8152bb84b9e628dd" is_pre_election: true
I20260620 23:00:59.393251 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.393827 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0150d2ce16de43f7b479d9beebd9f137 | 127.10.134.190:43173 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:34791 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
2b1834d241b74f0f8152bb84b9e628dd | 127.10.134.129:33119 | HEALTHY | <none> | 0 | 0
c707c04fc03d49da82f97968100c19d0 | 127.10.134.130:39755 | HEALTHY | <none> | 1 | 0
e49a8d52878a43aa83df884ff96e687e | 127.10.134.131:44497 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:33119
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:39755
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:44497
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:33119
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:39755
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:44497
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:34791 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260620 23:00:59.425143 10778 log_verifier.cc:126] Checking tablet c4447a4bda0d41f09b88af81b2668b51
I20260620 23:00:59.487246 10778 log_verifier.cc:177] Verified matching terms for 48 ops in tablet c4447a4bda0d41f09b88af81b2668b51
I20260620 23:00:59.520782 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 12368
I20260620 23:00:59.529596 12490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:59.691011 13067 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.691253 13067 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P 2b1834d241b74f0f8152bb84b9e628dd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.691537 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 12368
W20260620 23:00:59.725459 12514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.134.129:33119 (error 108) [suppressed 3 similar messages]
I20260620 23:00:59.726049 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 12499
W20260620 23:00:59.729874 12514 consensus_peers.cc:487] T c4447a4bda0d41f09b88af81b2668b51 P c707c04fc03d49da82f97968100c19d0 -> Peer 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119): Couldn't send request to peer 2b1834d241b74f0f8152bb84b9e628dd. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:33119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:00:59.734630 12621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:00:59.742781 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.742952 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.743621 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
W20260620 23:00:59.745721 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119): Network error: Client connection negotiation failed: client connection to 127.10.134.129:33119: connect: Connection refused (error 111)
W20260620 23:00:59.756891 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755): Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.006s (SENT)
I20260620 23:00:59.757061 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.757381 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.766228 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.766418 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.767074 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
W20260620 23:00:59.768781 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119): Network error: Client connection negotiation failed: client connection to 127.10.134.129:33119: connect: Connection refused (error 111)
W20260620 23:00:59.788055 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755): Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.010s (SENT)
I20260620 23:00:59.788231 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.788697 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.799857 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.800081 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.800803 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
W20260620 23:00:59.802609 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119): Network error: Client connection negotiation failed: client connection to 127.10.134.129:33119: connect: Connection refused (error 111)
W20260620 23:00:59.832250 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755): Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.015s (SENT)
I20260620 23:00:59.832460 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.832805 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.848263 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.848515 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.849346 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
W20260620 23:00:59.851217 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119): Network error: Client connection negotiation failed: client connection to 127.10.134.129:33119: connect: Connection refused (error 111)
W20260620 23:00:59.868835 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755): Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.009s (SENT)
I20260620 23:00:59.868997 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.869292 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.880328 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:00:59.880501 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:00:59.881099 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
W20260620 23:00:59.882886 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119): Network error: Client connection negotiation failed: client connection to 127.10.134.129:33119: connect: Connection refused (error 111)
I20260620 23:00:59.897981 12576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:59.898137 12574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4447a4bda0d41f09b88af81b2668b51" candidate_uuid: "e49a8d52878a43aa83df884ff96e687e" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "c707c04fc03d49da82f97968100c19d0" is_pre_election: true
I20260620 23:00:59.899104 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 12499
W20260620 23:00:59.906216 12644 leader_election.cc:334] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755): Timed out: RequestConsensusVote RPC to 127.10.134.130:39755 timed out after 0.012s (SENT)
I20260620 23:00:59.906358 12644 leader_election.cc:302] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49a8d52878a43aa83df884ff96e687e; no voters: 2b1834d241b74f0f8152bb84b9e628dd, c707c04fc03d49da82f97968100c19d0
I20260620 23:00:59.906638 13064 raft_consensus.cc:2726] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:00:59.923265 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 12629
I20260620 23:00:59.929056 12751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:01:00.101774 13064 raft_consensus.cc:482] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c707c04fc03d49da82f97968100c19d0)
I20260620 23:01:00.102066 13064 raft_consensus.cc:504] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c707c04fc03d49da82f97968100c19d0" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39755 } } peers { permanent_uuid: "2b1834d241b74f0f8152bb84b9e628dd" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 33119 } } peers { permanent_uuid: "e49a8d52878a43aa83df884ff96e687e" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 44497 } }
I20260620 23:01:00.102439 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 12629
I20260620 23:01:00.102739 13064 leader_election.cc:288] T c4447a4bda0d41f09b88af81b2668b51 P e49a8d52878a43aa83df884ff96e687e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c707c04fc03d49da82f97968100c19d0 (127.10.134.130:39755), 2b1834d241b74f0f8152bb84b9e628dd (127.10.134.129:33119)
I20260620 23:01:00.137004 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 12277
I20260620 23:01:00.138135 12338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:01:00.247524 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 12277
2026-06-20T23:01:00Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (37501 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20260620 23:01:00.291337 10778 test_util.cc:268] Using random seed: 1644279027
I20260620 23:01:00.293097 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:01:00.293180 10778 ts_itest-base.cc:113] --------------
I20260620 23:01:00.293247 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:01:00.293299 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:01:00.293363 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:00Z Disabled control of system clock
I20260620 23:01:00.319625 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:34675
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:34675 with env {}
W20260620 23:01:00.642118 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:00.642414 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:00.642489 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:00.649642 13090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:00.649750 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:00.649793 13090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:00.649842 13090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:00.658975 13090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:34675
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:00.660291 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:00.661918 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:00.673755 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:00.673846 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:00.673846 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:00.673966 13090 server_base.cc:1034] running on GCE node
I20260620 23:01:00.675263 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:00.676692 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:00.677901 13090 hybrid_clock.cc:648] HybridClock initialized: now 1781996460677849 us; error 36 us; skew 500 ppm
I20260620 23:01:00.678287 13090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:00.680533 13090 webserver.cc:460] Webserver started at http://127.10.134.190:38025/ using document root <none> and password file <none>
I20260620 23:01:00.681227 13090 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:00.681367 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:00.681681 13090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:00.684340 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "284d9dcd534240888dba87e55c1b2ebd"
format_stamp: "Formatted at 2026-06-20 23:01:00 on dist-test-slave-8k25"
I20260620 23:01:00.685079 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "284d9dcd534240888dba87e55c1b2ebd"
format_stamp: "Formatted at 2026-06-20 23:01:00 on dist-test-slave-8k25"
I20260620 23:01:00.690256 13090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 23:01:00.693754 13104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:00.695243 13090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:01:00.695430 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "284d9dcd534240888dba87e55c1b2ebd"
format_stamp: "Formatted at 2026-06-20 23:01:00 on dist-test-slave-8k25"
I20260620 23:01:00.695629 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:00.728798 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:00.729691 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:00.730027 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:00.747759 13090 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:34675
I20260620 23:01:00.747797 13155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:34675 every 8 connection(s)
I20260620 23:01:00.749079 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:01:00.752764 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:00.757563 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 13090
I20260620 23:01:00.757741 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:01:00.763476 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd: Bootstrap starting.
I20260620 23:01:00.766983 13156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:00.768203 13156 log.cc:824] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:00.770955 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd: No bootstrap required, opened a new log
I20260620 23:01:00.779187 13156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284d9dcd534240888dba87e55c1b2ebd" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 34675 } }
I20260620 23:01:00.779596 13156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:00.779703 13156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284d9dcd534240888dba87e55c1b2ebd, State: Initialized, Role: FOLLOWER
I20260620 23:01:00.780328 13156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284d9dcd534240888dba87e55c1b2ebd" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 34675 } }
I20260620 23:01:00.780555 13156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:00.780695 13156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:00.780866 13156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:00.782972 13156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284d9dcd534240888dba87e55c1b2ebd" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 34675 } }
I20260620 23:01:00.783465 13156 leader_election.cc:302] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 284d9dcd534240888dba87e55c1b2ebd; no voters:
I20260620 23:01:00.784111 13156 leader_election.cc:288] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:00.784299 13161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:00.786412 13156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:00.791072 13161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [term 1 LEADER]: Becoming Leader. State: Replica: 284d9dcd534240888dba87e55c1b2ebd, State: Running, Role: LEADER
I20260620 23:01:00.791769 13161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284d9dcd534240888dba87e55c1b2ebd" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 34675 } }
I20260620 23:01:00.795033 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284d9dcd534240888dba87e55c1b2ebd. Latest consensus state: current_term: 1 leader_uuid: "284d9dcd534240888dba87e55c1b2ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284d9dcd534240888dba87e55c1b2ebd" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 34675 } } }
I20260620 23:01:00.795397 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:00.795006 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284d9dcd534240888dba87e55c1b2ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284d9dcd534240888dba87e55c1b2ebd" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 34675 } } }
I20260620 23:01:00.796033 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:00.796659 13176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:00.801824 13176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:00.809988 13176 catalog_manager.cc:1347] Generated new cluster ID: f13aeb12ff494dcd91a707f99516787f
I20260620 23:01:00.810096 13176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:00.822654 13176 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:00.823945 13176 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:00.829816 13176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 284d9dcd534240888dba87e55c1b2ebd: Generated new TSK 0
I20260620 23:01:00.830708 13176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:00.835294 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:34675
--builtin_ntp_servers=127.10.134.148:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20260620 23:01:01.156692 13180 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260620 23:01:01.156996 13180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:01.157063 13180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:01.157135 13180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:01.164276 13180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:01.164422 13180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:01:01.173297 13180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:34675
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:01.174553 13180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:01.176225 13180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:01.189098 13186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:01.189196 13185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:01.189216 13188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:01.190508 13180 server_base.cc:1034] running on GCE node
I20260620 23:01:01.191095 13180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:01.192143 13180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:01.193361 13180 hybrid_clock.cc:648] HybridClock initialized: now 1781996461193308 us; error 56 us; skew 500 ppm
I20260620 23:01:01.193632 13180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:01.195823 13180 webserver.cc:460] Webserver started at http://127.10.134.129:33681/ using document root <none> and password file <none>
I20260620 23:01:01.196522 13180 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:01.196614 13180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:01.196872 13180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:01.199517 13180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6dce4229f6f247c7b2b68b3a75c39539"
format_stamp: "Formatted at 2026-06-20 23:01:01 on dist-test-slave-8k25"
I20260620 23:01:01.200326 13180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6dce4229f6f247c7b2b68b3a75c39539"
format_stamp: "Formatted at 2026-06-20 23:01:01 on dist-test-slave-8k25"
I20260620 23:01:01.205155 13180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:01:01.208393 13194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:01.209658 13180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:01.209777 13180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6dce4229f6f247c7b2b68b3a75c39539"
format_stamp: "Formatted at 2026-06-20 23:01:01 on dist-test-slave-8k25"
I20260620 23:01:01.210005 13180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:01.221390 13180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:01.222174 13180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:01.222460 13180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:01.223315 13180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:01.224601 13180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:01.224684 13180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:01.224756 13180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:01.224808 13180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:01.257522 13180 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:36713
I20260620 23:01:01.257571 13306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:36713 every 8 connection(s)
I20260620 23:01:01.258903 13180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:01:01.263409 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 13180
I20260620 23:01:01.263617 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:01:01.267582 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:34675
--builtin_ntp_servers=127.10.134.148:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260620 23:01:01.275471 13307 heartbeater.cc:347] Connected to a master server at 127.10.134.190:34675
I20260620 23:01:01.275817 13307 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:01.276722 13307 heartbeater.cc:511] Master 127.10.134.190:34675 requested a full tablet report, sending...
I20260620 23:01:01.279014 13121 ts_manager.cc:194] Registered new tserver with Master: 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:01.280509 13121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:50353
W20260620 23:01:01.591851 13310 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260620 23:01:01.592312 13310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:01.592402 13310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:01.592480 13310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:01.599575 13310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:01.599766 13310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:01:01.608685 13310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:34675
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:01.610002 13310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:01.611690 13310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:01.624125 13315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:01.624135 13316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:01.624171 13318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:01.624851 13310 server_base.cc:1034] running on GCE node
I20260620 23:01:01.625478 13310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:01.626436 13310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:01.627658 13310 hybrid_clock.cc:648] HybridClock initialized: now 1781996461627625 us; error 51 us; skew 500 ppm
I20260620 23:01:01.627982 13310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:01.630247 13310 webserver.cc:460] Webserver started at http://127.10.134.130:43741/ using document root <none> and password file <none>
I20260620 23:01:01.630924 13310 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:01.631052 13310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:01.631358 13310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:01.633855 13310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
format_stamp: "Formatted at 2026-06-20 23:01:01 on dist-test-slave-8k25"
I20260620 23:01:01.635047 13310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
format_stamp: "Formatted at 2026-06-20 23:01:01 on dist-test-slave-8k25"
I20260620 23:01:01.640081 13310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 23:01:01.643535 13324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:01.644850 13310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:01:01.645035 13310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
format_stamp: "Formatted at 2026-06-20 23:01:01 on dist-test-slave-8k25"
I20260620 23:01:01.645278 13310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:01.658656 13310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:01.659492 13310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:01.659780 13310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:01.660704 13310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:01.662199 13310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:01.662315 13310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:01.662451 13310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:01.662534 13310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:01.694808 13310 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:38257
I20260620 23:01:01.694859 13436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:38257 every 8 connection(s)
I20260620 23:01:01.696142 13310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:01:01.698694 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 13310
I20260620 23:01:01.698873 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:01:01.702272 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:34675
--builtin_ntp_servers=127.10.134.148:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260620 23:01:01.712083 13437 heartbeater.cc:347] Connected to a master server at 127.10.134.190:34675
I20260620 23:01:01.712417 13437 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:01.713227 13437 heartbeater.cc:511] Master 127.10.134.190:34675 requested a full tablet report, sending...
I20260620 23:01:01.715054 13121 ts_manager.cc:194] Registered new tserver with Master: 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:01.715852 13121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:54149
W20260620 23:01:02.024922 13440 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260620 23:01:02.025276 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:02.025386 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:02.025512 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:02.032936 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:02.033136 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:01:02.042317 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:34675
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:02.043632 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:02.045356 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:02.058424 13448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:02.058398 13446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:02.059103 13440 server_base.cc:1034] running on GCE node
W20260620 23:01:02.059127 13445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:02.059782 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:02.060679 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:02.061905 13440 hybrid_clock.cc:648] HybridClock initialized: now 1781996462061860 us; error 58 us; skew 500 ppm
I20260620 23:01:02.062269 13440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:02.064504 13440 webserver.cc:460] Webserver started at http://127.10.134.131:36507/ using document root <none> and password file <none>
I20260620 23:01:02.065196 13440 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:02.065325 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:02.065619 13440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:02.068184 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eb45f804a3ff4b12a653ee059501b6f5"
format_stamp: "Formatted at 2026-06-20 23:01:02 on dist-test-slave-8k25"
I20260620 23:01:02.068938 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eb45f804a3ff4b12a653ee059501b6f5"
format_stamp: "Formatted at 2026-06-20 23:01:02 on dist-test-slave-8k25"
I20260620 23:01:02.074412 13440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 23:01:02.077770 13454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:02.079185 13440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:01:02.079376 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eb45f804a3ff4b12a653ee059501b6f5"
format_stamp: "Formatted at 2026-06-20 23:01:02 on dist-test-slave-8k25"
I20260620 23:01:02.079597 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:02.102012 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:02.102873 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:02.103159 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:02.104094 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:02.105446 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:02.105569 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:02.105701 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:02.105786 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:02.137413 13440 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:40719
I20260620 23:01:02.137497 13566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:40719 every 8 connection(s)
I20260620 23:01:02.138715 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:01:02.143190 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 13440
I20260620 23:01:02.143378 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:01:02.153003 13567 heartbeater.cc:347] Connected to a master server at 127.10.134.190:34675
I20260620 23:01:02.153255 13567 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:02.153900 13567 heartbeater.cc:511] Master 127.10.134.190:34675 requested a full tablet report, sending...
I20260620 23:01:02.155191 13121 ts_manager.cc:194] Registered new tserver with Master: eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719)
I20260620 23:01:02.155903 13121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:50589
I20260620 23:01:02.158691 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:01:02.161736 10778 test_util.cc:268] Using random seed: 1646149411
I20260620 23:01:02.180092 13121 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37642:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:02.181624 13121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:02.222618 13502 tablet_service.cc:1467] Processing CreateTablet for tablet cb4f22b70c6e41e49ebfc29f973106da (DEFAULT_TABLE table=test-workload [id=a2a5ccb43a764745b7815fdefad32428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:02.222491 13242 tablet_service.cc:1467] Processing CreateTablet for tablet cb4f22b70c6e41e49ebfc29f973106da (DEFAULT_TABLE table=test-workload [id=a2a5ccb43a764745b7815fdefad32428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:02.224079 13242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4f22b70c6e41e49ebfc29f973106da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:02.224202 13502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4f22b70c6e41e49ebfc29f973106da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:02.225320 13372 tablet_service.cc:1467] Processing CreateTablet for tablet cb4f22b70c6e41e49ebfc29f973106da (DEFAULT_TABLE table=test-workload [id=a2a5ccb43a764745b7815fdefad32428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:02.226631 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4f22b70c6e41e49ebfc29f973106da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:02.234936 13585 tablet_bootstrap.cc:492] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5: Bootstrap starting.
I20260620 23:01:02.238651 13586 tablet_bootstrap.cc:492] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539: Bootstrap starting.
I20260620 23:01:02.239437 13585 tablet_bootstrap.cc:654] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:02.240072 13587 tablet_bootstrap.cc:492] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45: Bootstrap starting.
I20260620 23:01:02.240832 13585 log.cc:824] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:02.242161 13586 tablet_bootstrap.cc:654] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:02.243067 13587 tablet_bootstrap.cc:654] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:02.243335 13586 log.cc:824] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:02.243739 13585 tablet_bootstrap.cc:492] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5: No bootstrap required, opened a new log
I20260620 23:01:02.244038 13585 ts_tablet_manager.cc:1397] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:01:02.244143 13587 log.cc:824] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:02.246091 13586 tablet_bootstrap.cc:492] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539: No bootstrap required, opened a new log
I20260620 23:01:02.246340 13586 ts_tablet_manager.cc:1397] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260620 23:01:02.246645 13587 tablet_bootstrap.cc:492] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45: No bootstrap required, opened a new log
I20260620 23:01:02.246887 13587 ts_tablet_manager.cc:1397] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:01:02.254278 13587 raft_consensus.cc:348] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.254648 13587 raft_consensus.cc:374] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:02.254737 13587 raft_consensus.cc:729] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a46fb3b701f43cb96ea5c4958cb0a45, State: Initialized, Role: FOLLOWER
I20260620 23:01:02.254612 13585 raft_consensus.cc:348] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.255028 13585 raft_consensus.cc:374] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:02.255138 13585 raft_consensus.cc:729] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb45f804a3ff4b12a653ee059501b6f5, State: Initialized, Role: FOLLOWER
I20260620 23:01:02.255750 13587 consensus_queue.cc:260] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.256534 13591 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:02.257002 13586 raft_consensus.cc:348] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.257545 13586 raft_consensus.cc:374] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:02.257692 13586 raft_consensus.cc:729] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dce4229f6f247c7b2b68b3a75c39539, State: Initialized, Role: FOLLOWER
I20260620 23:01:02.258231 13591 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.258548 13586 consensus_queue.cc:260] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.263008 13585 consensus_queue.cc:260] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.263808 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:02.264135 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.269856 13307 heartbeater.cc:503] Master 127.10.134.190:34675 was elected leader, sending a full tablet report...
I20260620 23:01:02.271214 13597 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:02.271463 13597 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.273794 13591 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:02.274768 13437 heartbeater.cc:503] Master 127.10.134.190:34675 was elected leader, sending a full tablet report...
I20260620 23:01:02.276605 13567 heartbeater.cc:503] Master 127.10.134.190:34675 was elected leader, sending a full tablet report...
W20260620 23:01:02.276759 13456 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.130:38257 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
W20260620 23:01:02.277923 13328 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.129:36713 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260620 23:01:02.272974 13586 ts_tablet_manager.cc:1428] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539: Time spent starting tablet: real 0.027s user 0.017s sys 0.002s
I20260620 23:01:02.284178 13585 ts_tablet_manager.cc:1428] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5: Time spent starting tablet: real 0.040s user 0.008s sys 0.014s
I20260620 23:01:02.286072 13587 ts_tablet_manager.cc:1428] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45: Time spent starting tablet: real 0.039s user 0.008s sys 0.009s
I20260620 23:01:02.284085 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:02.290004 13392 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb45f804a3ff4b12a653ee059501b6f5 in term 0.
I20260620 23:01:02.290756 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:02.293496 13196 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.130:38257 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
W20260620 23:01:02.295905 13456 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257): Timed out: RequestConsensusVote RPC to 127.10.134.130:38257 timed out after 0.008s (SENT)
I20260620 23:01:02.298219 13597 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:02.302845 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:02.303112 13392 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dce4229f6f247c7b2b68b3a75c39539 in term 0.
I20260620 23:01:02.303192 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:02.303701 13522 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dce4229f6f247c7b2b68b3a75c39539 in term 0.
I20260620 23:01:02.303778 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539; no voters:
I20260620 23:01:02.304198 13597 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:02.304368 13597 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:02.304477 13597 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:02.305625 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.306824 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:02.307039 13522 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 0.
W20260620 23:01:02.307466 13458 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Timed out: RequestConsensusVote RPC to 127.10.134.129:36713 timed out after 0.008s (SENT)
I20260620 23:01:02.307601 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:02.307649 13597 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.307626 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.307984 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 23:01:02.308295 13325 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719): Timed out: RequestConsensusVote RPC to 127.10.134.131:40719 timed out after 0.007s (SENT)
I20260620 23:01:02.308640 13262 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in current term 1: Already voted for candidate 6dce4229f6f247c7b2b68b3a75c39539 in this term.
I20260620 23:01:02.308738 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5"
I20260620 23:01:02.308841 13597 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 1 election: Requested vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:02.308894 13522 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:01:02.309770 13328 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Timed out: RequestConsensusVote RPC to 127.10.134.129:36713 timed out after 0.007s (SENT)
I20260620 23:01:02.309782 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
I20260620 23:01:02.310071 13392 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:02.310029 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:02.311372 13522 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dce4229f6f247c7b2b68b3a75c39539 in term 1.
I20260620 23:01:02.311743 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5; no voters:
I20260620 23:01:02.312083 13597 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:02.312235 13597 raft_consensus.cc:686] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 LEADER]: Becoming Leader. State: Replica: 6dce4229f6f247c7b2b68b3a75c39539, State: Running, Role: LEADER
I20260620 23:01:02.312587 13392 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dce4229f6f247c7b2b68b3a75c39539 in term 1.
I20260620 23:01:02.312813 13593 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:01:02.312726 13597 consensus_queue.cc:237] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.317757 13121 catalog_manager.cc:5495] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129). New cstate: current_term: 1 leader_uuid: "6dce4229f6f247c7b2b68b3a75c39539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } health_report { overall_health: HEALTHY } } }
I20260620 23:01:02.318272 13607 consensus_queue.cc:1035] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:02.319593 13607 consensus_queue.cc:1035] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:02.319931 13593 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:02.320134 13593 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.320932 13593 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:02.321882 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.322338 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:02.323154 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:02.325309 13591 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:01:02.328169 13594 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:01:02.328279 13594 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260620 23:01:02.331525 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:02.331709 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.332417 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:02.334800 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.336781 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:02.337497 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:02.339519 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:02.340168 13591 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:02.340353 13591 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.340997 13591 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:02.341331 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:02.341832 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.342334 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:02.342705 13591 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:02.700294 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:02.700491 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.701148 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:02.701490 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:02.701643 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.702363 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:02.702739 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:02.753671 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:02.753865 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:02.754562 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:02.754873 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:02.755043 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:02.756712 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:02.757012 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:01:03.029974 13522 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:01:03.030108 13522 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260620 23:01:03.030395 13392 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:01:03.030483 13392 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260620 23:01:03.205830 13618 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:03.206091 13618 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:03.207132 13618 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:03.207612 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:03.207600 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:03.208158 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:03.208405 13618 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:01:03.233343 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.723s user 0.000s sys 0.000s
W20260620 23:01:03.233685 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.724s user 0.000s sys 0.000s
I20260620 23:01:03.581578 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:03.581791 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:03.582597 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:03.583190 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:03.583274 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:03.584331 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:03.584597 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:03.700809 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:03.701014 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:03.701778 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:03.702208 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:03.706895 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:03.707463 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:03.707728 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:04.298439 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:04.298662 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:04.299371 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:04.299893 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:04.299998 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:04.300743 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:04.301021 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:04.351095 13629 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=0, attempt_no=0}) took 1979 ms. Trace:
I20260620 23:01:04.351637 13629 rpcz_store.cc:276] 0620 23:01:02.371288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:02.371334 (+ 46us) service_pool.cc:227] Handling call
0620 23:01:04.351025 (+1979691us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:03.028375 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:03.028497 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:03.028506 (+ 9us) write_op.cc:435] Acquired schema lock
0620 23:01:03.028510 (+ 4us) tablet.cc:660] Decoding operations
0620 23:01:03.028603 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:03.028641 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:03.028647 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:03.028703 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:01:03.028705 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:03.028753 (+ 48us) write_op.cc:270] Start()
0620 23:01:03.890892 (+862139us) write_op.cc:276] Timestamp: P: 1781996463028828 usec, L: 0
0620 23:01:03.890897 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:03.891169 (+ 272us) log.cc:842] Serialized 345 byte log entry
0620 23:01:04.342644 (+451475us) op_driver.cc:444] REPLICATION: finished
0620 23:01:04.350310 (+ 7666us) write_op.cc:301] APPLY: starting
0620 23:01:04.350336 (+ 26us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:04.350387 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:04.350411 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:04.350510 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:04.350538 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:04.350542 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:04.350693 (+ 151us) log.cc:842] Serialized 22 byte log entry
0620 23:01:04.350857 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:04.350877 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:04.350884 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":2309120,"child_traces":[["op",{"apply.queue_time_us":7567,"mutex_wait_us":22,"num_ops":1,"prepare.queue_time_us":652905,"prepare.run_cpu_time_us":839,"prepare.run_wall_time_us":862906,"raft.queue_time_us":102,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":495,"replication_time_us":451718,"spinlock_wait_cycles":35840,"thread_start_us":7435,"threads_started":1}]]}
W20260620 23:01:04.353319 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.969s user 0.001s sys 0.000s
W20260620 23:01:04.353343 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.969s user 0.001s sys 0.000s
I20260620 23:01:04.507781 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:04.508042 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:04.508898 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:04.509328 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:04.509364 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:04.510217 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:04.510641 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:04.599233 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:04.599469 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:04.600322 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:04.600811 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:04.600818 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:04.601481 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:04.621459 13629 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=2, attempt_no=0}) took 1392 ms. Trace:
I20260620 23:01:04.621641 13629 rpcz_store.cc:276] 0620 23:01:03.229271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:03.229337 (+ 66us) service_pool.cc:227] Handling call
0620 23:01:04.621381 (+1392044us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:03.891331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:03.891456 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:03.891462 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:03.891465 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:03.891570 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:03.891581 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:03.891586 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:03.891643 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:01:03.891645 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:03.891689 (+ 44us) write_op.cc:270] Start()
0620 23:01:04.545074 (+653385us) write_op.cc:276] Timestamp: P: 1781996463891718 usec, L: 0
0620 23:01:04.545079 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:04.545384 (+ 305us) log.cc:842] Serialized 354 byte log entry
0620 23:01:04.620701 (+ 75317us) op_driver.cc:444] REPLICATION: finished
0620 23:01:04.620826 (+ 125us) write_op.cc:301] APPLY: starting
0620 23:01:04.620844 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:04.620881 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:04.620913 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:04.620979 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:04.621012 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:04.621016 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:04.621097 (+ 81us) log.cc:842] Serialized 22 byte log entry
0620 23:01:04.621246 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:04.621264 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:04.621271 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":1,"prepare.queue_time_us":661456,"prepare.run_cpu_time_us":830,"prepare.run_wall_time_us":654124,"replication_time_us":75590}]]}
I20260620 23:01:04.912791 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:04.913022 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:04.913816 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:04.914345 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:04.914443 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:04.915035 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:05.088344 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:05.585786 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:05.761744 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:05.762017 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:05.762840 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:05.763388 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:05.763394 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:05.764058 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:05.764320 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:05.767351 13637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=4, attempt_no=0}) took 1143 ms. Trace:
I20260620 23:01:05.767479 13637 rpcz_store.cc:276] 0620 23:01:04.623813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:04.623876 (+ 63us) service_pool.cc:227] Handling call
0620 23:01:05.767287 (+1143411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:04.624274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:04.624357 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:04.624361 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:04.624362 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:04.624426 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:04.624435 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:04.624438 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:04.624473 (+ 35us) tablet.cc:699] Row locks acquired
0620 23:01:04.624475 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:04.624504 (+ 29us) write_op.cc:270] Start()
0620 23:01:05.475642 (+851138us) write_op.cc:276] Timestamp: P: 1781996464624585 usec, L: 0
0620 23:01:05.475646 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:05.475873 (+ 227us) log.cc:842] Serialized 354 byte log entry
0620 23:01:05.766295 (+290422us) op_driver.cc:444] REPLICATION: finished
0620 23:01:05.766725 (+ 430us) write_op.cc:301] APPLY: starting
0620 23:01:05.766743 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:05.766771 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:05.766881 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:05.766972 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:05.766997 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:05.767002 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:05.767064 (+ 62us) log.cc:842] Serialized 22 byte log entry
0620 23:01:05.767170 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:05.767185 (+ 15us) write_op.cc:454] Released schema lock
0620 23:01:05.767191 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":368,"num_ops":1,"prepare.queue_time_us":34,"prepare.run_cpu_time_us":631,"prepare.run_wall_time_us":851699,"raft.queue_time_us":50,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":170,"replication_time_us":290623,"spinlock_wait_cycles":3840,"thread_start_us":253,"threads_started":1}]]}
W20260620 23:01:05.769793 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.996s user 0.001s sys 0.000s
W20260620 23:01:05.769853 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.996s user 0.001s sys 0.000s
I20260620 23:01:06.050844 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:06.051057 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:06.051858 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:06.052345 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:06.052397 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:06.053043 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:06.385773 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:06.386035 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:06.386891 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:06.387374 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:06.387459 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:06.388023 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:06.388314 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:06.400897 13643 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=5, attempt_no=0}) took 1273 ms. Trace:
I20260620 23:01:06.401048 13643 rpcz_store.cc:276] 0620 23:01:05.127733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:05.127813 (+ 80us) service_pool.cc:227] Handling call
0620 23:01:06.400838 (+1273025us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:05.476011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:05.476148 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:05.476155 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:05.476158 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:05.476260 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:05.476300 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:05.476306 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:05.476372 (+ 66us) tablet.cc:699] Row locks acquired
0620 23:01:05.476376 (+ 4us) write_op.cc:260] PREPARE: finished
0620 23:01:05.476423 (+ 47us) write_op.cc:270] Start()
0620 23:01:05.841565 (+365142us) write_op.cc:276] Timestamp: P: 1781996465476433 usec, L: 0
0620 23:01:05.841570 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:05.841812 (+ 242us) log.cc:842] Serialized 354 byte log entry
0620 23:01:06.399629 (+557817us) op_driver.cc:444] REPLICATION: finished
0620 23:01:06.400254 (+ 625us) write_op.cc:301] APPLY: starting
0620 23:01:06.400274 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:06.400319 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:06.400351 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:06.400441 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:06.400480 (+ 39us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:06.400490 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:01:06.400590 (+ 100us) log.cc:842] Serialized 22 byte log entry
0620 23:01:06.400704 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:06.400727 (+ 23us) write_op.cc:454] Released schema lock
0620 23:01:06.400737 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":554,"num_ops":1,"prepare.queue_time_us":347692,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":365885,"replication_time_us":558024,"thread_start_us":315,"threads_started":1}]]}
I20260620 23:01:06.692368 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:06.993085 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:06.993294 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:06.994194 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:06.994648 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:06.995268 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:06.995868 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:06.996165 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:07.023849 13648 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=6, attempt_no=0}) took 1254 ms. Trace:
I20260620 23:01:07.024055 13648 rpcz_store.cc:276] 0620 23:01:05.769679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:05.769788 (+ 109us) service_pool.cc:227] Handling call
0620 23:01:07.023784 (+1253996us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:05.841966 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:05.842101 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:05.842107 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:05.842110 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:05.842209 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:05.842221 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:05.842226 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:05.842337 (+ 111us) tablet.cc:699] Row locks acquired
0620 23:01:05.842341 (+ 4us) write_op.cc:260] PREPARE: finished
0620 23:01:05.842392 (+ 51us) write_op.cc:270] Start()
0620 23:01:06.804549 (+962157us) write_op.cc:276] Timestamp: P: 1781996465842357 usec, L: 0
0620 23:01:06.804554 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:06.804861 (+ 307us) log.cc:842] Serialized 354 byte log entry
0620 23:01:07.022652 (+217791us) op_driver.cc:444] REPLICATION: finished
0620 23:01:07.023127 (+ 475us) write_op.cc:301] APPLY: starting
0620 23:01:07.023147 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:07.023192 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:07.023258 (+ 66us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:07.023367 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:07.023402 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:07.023409 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:07.023523 (+ 114us) log.cc:842] Serialized 22 byte log entry
0620 23:01:07.023660 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:07.023678 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:07.023686 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":395,"num_ops":1,"prepare.queue_time_us":71604,"prepare.run_cpu_time_us":949,"prepare.run_wall_time_us":963023,"replication_time_us":218071,"thread_start_us":256,"threads_started":1}]]}
I20260620 23:01:07.025496 13221 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49000 (request call id 15) took 1104 ms. Trace:
I20260620 23:01:07.025638 13221 rpcz_store.cc:276] 0620 23:01:05.921265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:05.921352 (+ 87us) service_pool.cc:227] Handling call
0620 23:01:05.921611 (+ 259us) tablet_service.cc:2780] Created scanner 50a76abd60ea4ff7963f6a8c81b32bc5 for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is e629b913e47b438a978940936cbfa370
0620 23:01:05.922459 (+ 848us) tablet_service.cc:2920] Creating iterator
0620 23:01:05.922508 (+ 49us) tablet_service.cc:3297] Waiting safe time to advance
0620 23:01:06.805403 (+882895us) tablet_service.cc:3304] Waiting for operations to commit
0620 23:01:07.023700 (+218297us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1101177 microseconds
0620 23:01:07.023805 (+ 105us) tablet_service.cc:2945] Iterator created
0620 23:01:07.024374 (+ 569us) tablet_service.cc:2967] Iterator init: OK
0620 23:01:07.024404 (+ 30us) tablet_service.cc:3010] has_more: true
0620 23:01:07.024445 (+ 41us) tablet_service.cc:3027] Continuing scan request
0620 23:01:07.024501 (+ 56us) tablet_service.cc:3091] Found scanner 50a76abd60ea4ff7963f6a8c81b32bc5 for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is e629b913e47b438a978940936cbfa370
0620 23:01:07.025436 (+ 935us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":30336}
W20260620 23:01:07.026520 13615 scanner-internal.cc:458] Time spent opening tablet: real 1.106s user 0.001s sys 0.000s
I20260620 23:01:07.026892 13222 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49000 (request call id 16) took 1105 ms. Trace:
I20260620 23:01:07.027009 13222 rpcz_store.cc:276] 0620 23:01:05.921412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:05.921480 (+ 68us) service_pool.cc:227] Handling call
0620 23:01:05.921643 (+ 163us) tablet_service.cc:2780] Created scanner c1e28f89d6e5407d817c4b89da7e9a85 for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is c2d3b74466554573a75f06ea3e65db73
0620 23:01:05.922455 (+ 812us) tablet_service.cc:2920] Creating iterator
0620 23:01:05.922511 (+ 56us) tablet_service.cc:3297] Waiting safe time to advance
0620 23:01:06.804716 (+882205us) tablet_service.cc:3304] Waiting for operations to commit
0620 23:01:07.025344 (+220628us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1102817 microseconds
0620 23:01:07.025417 (+ 73us) tablet_service.cc:2945] Iterator created
0620 23:01:07.025861 (+ 444us) tablet_service.cc:2967] Iterator init: OK
0620 23:01:07.025888 (+ 27us) tablet_service.cc:3010] has_more: true
0620 23:01:07.025927 (+ 39us) tablet_service.cc:3027] Continuing scan request
0620 23:01:07.026029 (+ 102us) tablet_service.cc:3091] Found scanner c1e28f89d6e5407d817c4b89da7e9a85 for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is c2d3b74466554573a75f06ea3e65db73
0620 23:01:07.026819 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":35072}
W20260620 23:01:07.027642 13616 scanner-internal.cc:458] Time spent opening tablet: real 1.107s user 0.001s sys 0.000s
I20260620 23:01:07.104678 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:07.104907 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:07.105747 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:07.106112 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:07.106379 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:07.106973 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:07.107250 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:07.263319 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:07.263533 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:07.264212 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:07.264796 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:07.264951 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:07.265470 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:07.265897 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:07.645540 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:07.645800 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:07.646595 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:07.647114 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:07.647121 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:07.647903 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:07.696897 13652 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=7, attempt_no=0}) took 1293 ms. Trace:
I20260620 23:01:07.697118 13652 rpcz_store.cc:276] 0620 23:01:06.402914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:06.402954 (+ 40us) service_pool.cc:227] Handling call
0620 23:01:07.696814 (+1293860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:06.804997 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:06.805119 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:06.805125 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:06.805128 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:06.805231 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:06.805244 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:06.805252 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:06.805312 (+ 60us) tablet.cc:699] Row locks acquired
0620 23:01:06.805314 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:06.805358 (+ 44us) write_op.cc:270] Start()
0620 23:01:07.534510 (+729152us) write_op.cc:276] Timestamp: P: 1781996466805470 usec, L: 0
0620 23:01:07.534518 (+ 8us) op_driver.cc:331] REPLICATION: starting
0620 23:01:07.534822 (+ 304us) log.cc:842] Serialized 354 byte log entry
0620 23:01:07.694779 (+159957us) op_driver.cc:444] REPLICATION: finished
0620 23:01:07.695357 (+ 578us) write_op.cc:301] APPLY: starting
0620 23:01:07.695375 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:07.695410 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:07.695440 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:07.695510 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:07.695537 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:07.695541 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:07.695650 (+ 109us) log.cc:842] Serialized 22 byte log entry
0620 23:01:07.696647 (+ 997us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:07.696675 (+ 28us) write_op.cc:454] Released schema lock
0620 23:01:07.696683 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":500,"num_ops":1,"prepare.queue_time_us":401609,"prepare.run_cpu_time_us":1000,"prepare.run_wall_time_us":730148,"raft.queue_time_us":91,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"replication_time_us":160222,"spinlock_wait_cycles":612096,"thread_start_us":354,"threads_started":1}]]}
W20260620 23:01:07.698539 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.521s user 0.001s sys 0.000s
W20260620 23:01:07.698592 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.520s user 0.001s sys 0.000s
I20260620 23:01:07.947302 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:08.382567 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:08.382797 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:08.383651 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:08.384142 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:08.384141 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:08.384806 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:08.741392 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:08.741621 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:08.742486 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:08.743021 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:08.743028 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:08.743724 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:08.770210 13659 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=9, attempt_no=0}) took 1070 ms. Trace:
I20260620 23:01:08.770493 13659 rpcz_store.cc:276] 0620 23:01:07.699491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:07.699560 (+ 69us) service_pool.cc:227] Handling call
0620 23:01:08.770133 (+1070573us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:07.700022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:07.700132 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:07.700137 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:07.700140 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:07.700275 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:07.700289 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:07.700296 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:07.700353 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:01:07.700356 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:07.700392 (+ 36us) write_op.cc:270] Start()
0620 23:01:07.845561 (+145169us) write_op.cc:276] Timestamp: P: 1781996467700475 usec, L: 0
0620 23:01:07.845566 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:07.845826 (+ 260us) log.cc:842] Serialized 354 byte log entry
0620 23:01:08.769044 (+923218us) op_driver.cc:444] REPLICATION: finished
0620 23:01:08.769508 (+ 464us) write_op.cc:301] APPLY: starting
0620 23:01:08.769525 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:08.769560 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:08.769587 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:08.769659 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:08.769685 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:08.769689 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:08.769775 (+ 86us) log.cc:842] Serialized 22 byte log entry
0620 23:01:08.769902 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:08.769923 (+ 21us) write_op.cc:454] Released schema lock
0620 23:01:08.769930 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":393,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":828,"prepare.run_wall_time_us":145898,"replication_time_us":923440,"thread_start_us":239,"threads_started":1}]]}
W20260620 23:01:08.772213 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.923s user 0.001s sys 0.000s
W20260620 23:01:08.772312 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.923s user 0.001s sys 0.000s
I20260620 23:01:09.445405 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:09.473116 13666 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=10, attempt_no=0}) took 1502 ms. Trace:
I20260620 23:01:09.473245 13666 rpcz_store.cc:276] 0620 23:01:07.970564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:07.970636 (+ 72us) service_pool.cc:227] Handling call
0620 23:01:09.473055 (+1502419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:07.971099 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:07.971260 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:07.971264 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:07.971266 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:07.971335 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:07.971344 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:07.971347 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:07.971394 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:07.971395 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:07.971426 (+ 31us) write_op.cc:270] Start()
0620 23:01:08.665616 (+694190us) write_op.cc:276] Timestamp: P: 1781996467971433 usec, L: 0
0620 23:01:08.665622 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:08.665903 (+ 281us) log.cc:842] Serialized 354 byte log entry
0620 23:01:09.472034 (+806131us) op_driver.cc:444] REPLICATION: finished
0620 23:01:09.472490 (+ 456us) write_op.cc:301] APPLY: starting
0620 23:01:09.472507 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:09.472541 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:09.472566 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:09.472642 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:09.472669 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:09.472673 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:09.472762 (+ 89us) log.cc:842] Serialized 22 byte log entry
0620 23:01:09.472830 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:09.472846 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:09.472851 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":390,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":847,"prepare.run_wall_time_us":694917,"replication_time_us":806378,"thread_start_us":265,"threads_started":1}]]}
I20260620 23:01:09.901027 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:09.901232 13391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.129:55473 (request call id 1107) took 1516 ms. Trace:
I20260620 23:01:09.901396 13391 rpcz_store.cc:276] 0620 23:01:08.384425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:08.384475 (+ 50us) service_pool.cc:227] Handling call
0620 23:01:08.384691 (+ 216us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:08.384865 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 9
0620 23:01:08.385209 (+ 344us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:08.385335 (+ 126us) write_op.cc:270] Start()
0620 23:01:09.027518 (+642183us) write_op.cc:276] Timestamp: P: 1781996467535755 usec, L: 0
0620 23:01:09.028282 (+ 764us) write_op.cc:270] Start()
0620 23:01:09.899449 (+871167us) write_op.cc:276] Timestamp: P: 1781996467700475 usec, L: 0
0620 23:01:09.900118 (+ 669us) log.cc:842] Serialized 708 byte log entry
0620 23:01:09.900520 (+ 402us) raft_consensus.cc:1631] Marking committed up to 10
0620 23:01:09.900891 (+ 371us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:09.900932 (+ 41us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:09.901111 (+ 179us) raft_consensus.cc:1671] finished
0620 23:01:09.901114 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:09.901174 (+ 60us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:09.028182 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:09.028302 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:09.028307 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:09.028308 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:09.028465 (+ 157us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:09.028486 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:09.028493 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:09.028556 (+ 63us) tablet.cc:699] Row locks acquired
0620 23:01:09.028558 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:09.900575 (+872017us) op_driver.cc:444] REPLICATION: finished
0620 23:01:09.900966 (+ 391us) write_op.cc:301] APPLY: starting
0620 23:01:09.900982 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:09.901029 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:09.901031 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:09.901102 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:09.901137 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:09.901201 (+ 64us) write_op.cc:312] APPLY: finished
0620 23:01:09.901433 (+ 232us) log.cc:842] Serialized 22 byte log entry
0620 23:01:09.901524 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:09.901540 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:09.901547 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:09.901636 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:09.901640 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:09.899987 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:09.900094 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:09.900099 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:09.900101 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:09.900318 (+ 217us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:09.900355 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:09.900359 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:09.900414 (+ 55us) tablet.cc:699] Row locks acquired
0620 23:01:09.900415 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":264,"threads_started":1,"wal-append.queue_time_us":377,"child_traces":[["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":679,"apply.run_wall_time_us":748,"num_ops":1,"partition_lock_wait_us":9,"prepare.queue_time_us":516,"prepare.run_cpu_time_us":475,"prepare.run_wall_time_us":475,"replication_time_us":1515217,"thread_start_us":653,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":30,"prepare.queue_time_us":403,"prepare.run_cpu_time_us":527,"prepare.run_wall_time_us":573,"thread_start_us":276,"threads_started":1}]]}
I20260620 23:01:10.313371 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:10.313612 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:10.314560 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:10.315016 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:10.315202 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:10.315863 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:10.316355 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:10.357347 13673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=11, attempt_no=0}) took 1584 ms. Trace:
I20260620 23:01:10.357532 13673 rpcz_store.cc:276] 0620 23:01:08.772821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:08.772889 (+ 68us) service_pool.cc:227] Handling call
0620 23:01:10.357281 (+1584392us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:08.773358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:08.773480 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:08.773486 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:08.773488 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:08.773576 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:08.773584 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:08.773588 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:08.773626 (+ 38us) tablet.cc:699] Row locks acquired
0620 23:01:08.773627 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:08.773654 (+ 27us) write_op.cc:270] Start()
0620 23:01:09.308833 (+535179us) write_op.cc:276] Timestamp: P: 1781996468773670 usec, L: 0
0620 23:01:09.308838 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:09.309234 (+ 396us) log.cc:842] Serialized 354 byte log entry
0620 23:01:10.355900 (+1046666us) op_driver.cc:444] REPLICATION: finished
0620 23:01:10.356540 (+ 640us) write_op.cc:301] APPLY: starting
0620 23:01:10.356561 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:10.356612 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:10.356695 (+ 83us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:10.356776 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:10.356811 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:10.356817 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:10.356902 (+ 85us) log.cc:842] Serialized 22 byte log entry
0620 23:01:10.357108 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:10.357129 (+ 21us) write_op.cc:454] Released schema lock
0620 23:01:10.357136 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":574,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":892,"prepare.run_wall_time_us":536018,"replication_time_us":1047030,"thread_start_us":454,"threads_started":1}]]}
I20260620 23:01:10.359067 13222 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49000 (request call id 27) took 1435 ms. Trace:
I20260620 23:01:10.359200 13222 rpcz_store.cc:276] 0620 23:01:08.923940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:08.924007 (+ 67us) service_pool.cc:227] Handling call
0620 23:01:08.924221 (+ 214us) tablet_service.cc:2780] Created scanner 830fc95596234c978a0465fb3b0b5d2c for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is ea05eda5e6e54180b4b8e1f33a69f909
0620 23:01:08.924974 (+ 753us) tablet_service.cc:2920] Creating iterator
0620 23:01:08.925025 (+ 51us) tablet_service.cc:3297] Waiting safe time to advance
0620 23:01:09.309009 (+383984us) tablet_service.cc:3304] Waiting for operations to commit
0620 23:01:10.357134 (+1048125us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1432091 microseconds
0620 23:01:10.357293 (+ 159us) tablet_service.cc:2945] Iterator created
0620 23:01:10.357890 (+ 597us) tablet_service.cc:2967] Iterator init: OK
0620 23:01:10.357928 (+ 38us) tablet_service.cc:3010] has_more: true
0620 23:01:10.357996 (+ 68us) tablet_service.cc:3027] Continuing scan request
0620 23:01:10.358102 (+ 106us) tablet_service.cc:3091] Found scanner 830fc95596234c978a0465fb3b0b5d2c for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is ea05eda5e6e54180b4b8e1f33a69f909
0620 23:01:10.359014 (+ 912us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":24960}
I20260620 23:01:10.359694 13221 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49000 (request call id 26) took 1435 ms. Trace:
I20260620 23:01:10.359822 13221 rpcz_store.cc:276] 0620 23:01:08.923809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:08.923896 (+ 87us) service_pool.cc:227] Handling call
0620 23:01:08.924121 (+ 225us) tablet_service.cc:2780] Created scanner f1b8792dd1ef4bffbc232090589bd82e for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is aecfc1a75da94ee7a44c1e1d8bc253e9
0620 23:01:08.924989 (+ 868us) tablet_service.cc:2920] Creating iterator
0620 23:01:08.925028 (+ 39us) tablet_service.cc:3297] Waiting safe time to advance
0620 23:01:09.309027 (+383999us) tablet_service.cc:3304] Waiting for operations to commit
0620 23:01:10.358027 (+1049000us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1432985 microseconds
0620 23:01:10.358155 (+ 128us) tablet_service.cc:2945] Iterator created
0620 23:01:10.358650 (+ 495us) tablet_service.cc:2967] Iterator init: OK
0620 23:01:10.358676 (+ 26us) tablet_service.cc:3010] has_more: true
0620 23:01:10.358703 (+ 27us) tablet_service.cc:3027] Continuing scan request
0620 23:01:10.358799 (+ 96us) tablet_service.cc:3091] Found scanner f1b8792dd1ef4bffbc232090589bd82e for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is aecfc1a75da94ee7a44c1e1d8bc253e9
0620 23:01:10.359644 (+ 845us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":4736}
W20260620 23:01:10.360157 13616 scanner-internal.cc:458] Time spent opening tablet: real 1.437s user 0.001s sys 0.000s
W20260620 23:01:10.360558 13615 scanner-internal.cc:458] Time spent opening tablet: real 1.438s user 0.001s sys 0.000s
I20260620 23:01:10.638763 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:10.638994 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:10.639896 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:10.640467 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:10.640491 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:10.641229 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:10.641470 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:10.680755 13673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=12, attempt_no=0}) took 1205 ms. Trace:
I20260620 23:01:10.680974 13673 rpcz_store.cc:276] 0620 23:01:09.475078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:09.475151 (+ 73us) service_pool.cc:227] Handling call
0620 23:01:10.680703 (+1205552us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:09.475690 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:09.475783 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:09.475793 (+ 10us) write_op.cc:435] Acquired schema lock
0620 23:01:09.475795 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:09.475864 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:09.475872 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:09.475875 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:09.475913 (+ 38us) tablet.cc:699] Row locks acquired
0620 23:01:09.475914 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:09.475946 (+ 32us) write_op.cc:270] Start()
0620 23:01:10.394115 (+918169us) write_op.cc:276] Timestamp: P: 1781996469475930 usec, L: 0
0620 23:01:10.394121 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:10.394344 (+ 223us) log.cc:842] Serialized 354 byte log entry
0620 23:01:10.680025 (+285681us) op_driver.cc:444] REPLICATION: finished
0620 23:01:10.680138 (+ 113us) write_op.cc:301] APPLY: starting
0620 23:01:10.680163 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:10.680209 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:10.680241 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:10.680321 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:10.680359 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:10.680368 (+ 9us) write_op.cc:312] APPLY: finished
0620 23:01:10.680487 (+ 119us) log.cc:842] Serialized 22 byte log entry
0620 23:01:10.680598 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:10.680616 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:10.680623 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":661,"prepare.run_wall_time_us":918745,"replication_time_us":285875}]]}
I20260620 23:01:11.294304 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:11.294538 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:11.295311 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:11.295821 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:11.295838 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:11.296509 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:11.296779 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:11.318302 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:11.318353 13391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.129:55473 (request call id 1108) took 1414 ms. Trace:
I20260620 23:01:11.318524 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:11.319360 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:11.318527 13391 rpcz_store.cc:276] 0620 23:01:09.903782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:09.903842 (+ 60us) service_pool.cc:227] Handling call
0620 23:01:09.904012 (+ 170us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:09.904116 (+ 104us) raft_consensus.cc:1520] Early marking committed up to index 11
0620 23:01:09.904191 (+ 75us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:09.904275 (+ 84us) write_op.cc:270] Start()
0620 23:01:10.711438 (+807163us) write_op.cc:276] Timestamp: P: 1781996467971433 usec, L: 0
0620 23:01:10.711999 (+ 561us) write_op.cc:270] Start()
0620 23:01:11.315186 (+603187us) write_op.cc:276] Timestamp: P: 1781996468773670 usec, L: 0
0620 23:01:11.316256 (+ 1070us) log.cc:842] Serialized 708 byte log entry
0620 23:01:11.316565 (+ 309us) raft_consensus.cc:1631] Marking committed up to 12
0620 23:01:11.318173 (+ 1608us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:11.318213 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:11.318223 (+ 10us) raft_consensus.cc:1671] finished
0620 23:01:11.318225 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:11.318301 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:10.711897 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:10.712070 (+ 173us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:10.712074 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:10.712076 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:10.712152 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:10.712187 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:10.712193 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:10.712250 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:01:10.712254 (+ 4us) write_op.cc:260] PREPARE: finished
0620 23:01:11.316599 (+604345us) op_driver.cc:444] REPLICATION: finished
0620 23:01:11.318276 (+ 1677us) write_op.cc:301] APPLY: starting
0620 23:01:11.318300 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:11.318346 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:11.318350 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:11.318418 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:11.318451 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:11.318486 (+ 35us) write_op.cc:312] APPLY: finished
0620 23:01:11.318598 (+ 112us) log.cc:842] Serialized 22 byte log entry
0620 23:01:11.318659 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:11.318675 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:11.318682 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:11.318779 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:11.318783 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:11.316072 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:11.316204 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:11.316210 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:11.316212 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:11.316311 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:11.316325 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:11.316331 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:11.316399 (+ 68us) tablet.cc:699] Row locks acquired
0620 23:01:11.316404 (+ 5us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":221,"threads_started":1,"wal-append.queue_time_us":319,"child_traces":[["op",{"apply.queue_time_us":1587,"apply.run_cpu_time_us":590,"apply.run_wall_time_us":868,"num_ops":1,"partition_lock_wait_us":24,"prepare.queue_time_us":334,"prepare.run_cpu_time_us":469,"prepare.run_wall_time_us":469,"replication_time_us":1412310,"thread_start_us":1709,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":747,"prepare.run_cpu_time_us":456,"prepare.run_wall_time_us":455,"thread_start_us":641,"threads_started":1}]]}
I20260620 23:01:11.319684 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:11.320185 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:11.321039 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
W20260620 23:01:11.323557 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.812s user 0.001s sys 0.000s
W20260620 23:01:11.324853 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.814s user 0.001s sys 0.000s
I20260620 23:01:12.308840 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:12.309082 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:12.309803 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:12.310568 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:12.310546 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:12.311221 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:12.311494 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:12.357641 13690 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=14, attempt_no=0}) took 1674 ms. Trace:
I20260620 23:01:12.357908 13690 rpcz_store.cc:276] 0620 23:01:10.682814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:10.682907 (+ 93us) service_pool.cc:227] Handling call
0620 23:01:12.357566 (+1674659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:11.004471 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:11.004604 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:11.004610 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:11.004618 (+ 8us) tablet.cc:660] Decoding operations
0620 23:01:11.004696 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:11.004705 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:11.004708 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:11.004783 (+ 75us) tablet.cc:699] Row locks acquired
0620 23:01:11.004787 (+ 4us) write_op.cc:260] PREPARE: finished
0620 23:01:11.004835 (+ 48us) write_op.cc:270] Start()
0620 23:01:11.639975 (+635140us) write_op.cc:276] Timestamp: P: 1781996471004824 usec, L: 0
0620 23:01:11.639979 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:11.640273 (+ 294us) log.cc:842] Serialized 354 byte log entry
0620 23:01:12.356250 (+715977us) op_driver.cc:444] REPLICATION: finished
0620 23:01:12.356872 (+ 622us) write_op.cc:301] APPLY: starting
0620 23:01:12.356890 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:12.356926 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:12.356957 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:12.357131 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:12.357159 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:12.357163 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:12.357263 (+ 100us) log.cc:842] Serialized 22 byte log entry
0620 23:01:12.357387 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:12.357407 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:12.357418 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":551,"num_ops":1,"prepare.queue_time_us":321074,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":635891,"replication_time_us":716232,"thread_start_us":369,"threads_started":1}]]}
W20260620 23:01:12.360111 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.885s user 0.001s sys 0.000s
W20260620 23:01:12.360136 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.886s user 0.001s sys 0.000s
I20260620 23:01:12.722875 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:12.722884 13391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.129:55473 (request call id 1109) took 1402 ms. Trace:
I20260620 23:01:12.723217 13391 rpcz_store.cc:276] 0620 23:01:11.320806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:11.320890 (+ 84us) service_pool.cc:227] Handling call
0620 23:01:11.321146 (+ 256us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:11.321273 (+ 127us) raft_consensus.cc:1520] Early marking committed up to index 13
0620 23:01:11.321357 (+ 84us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:11.321492 (+ 135us) write_op.cc:270] Start()
0620 23:01:12.045658 (+724166us) write_op.cc:276] Timestamp: P: 1781996469475930 usec, L: 0
0620 23:01:12.046418 (+ 760us) write_op.cc:270] Start()
0620 23:01:12.720729 (+674311us) write_op.cc:276] Timestamp: P: 1781996470394933 usec, L: 0
0620 23:01:12.721398 (+ 669us) log.cc:842] Serialized 708 byte log entry
0620 23:01:12.721763 (+ 365us) raft_consensus.cc:1631] Marking committed up to 15
0620 23:01:12.722687 (+ 924us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:12.722723 (+ 36us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:12.722739 (+ 16us) raft_consensus.cc:1671] finished
0620 23:01:12.722741 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:12.722833 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:12.046340 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:12.046444 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:12.046449 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:12.046450 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:12.046531 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:12.046545 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:12.046549 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:12.046596 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:12.046597 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:12.721813 (+675216us) op_driver.cc:444] REPLICATION: finished
0620 23:01:12.722227 (+ 414us) write_op.cc:301] APPLY: starting
0620 23:01:12.722241 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:12.722274 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:12.722276 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:12.722349 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:12.722373 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:12.722408 (+ 35us) write_op.cc:312] APPLY: finished
0620 23:01:12.722489 (+ 81us) log.cc:842] Serialized 22 byte log entry
0620 23:01:12.722584 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:12.722598 (+ 14us) write_op.cc:454] Released schema lock
0620 23:01:12.722604 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:12.722685 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:12.722693 (+ 8us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:12.721214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:12.721319 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:12.721325 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:12.721333 (+ 8us) tablet.cc:660] Decoding operations
0620 23:01:12.721423 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:12.721435 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:12.721438 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:12.721485 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:12.721486 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:12.722162 (+ 676us) op_driver.cc:444] REPLICATION: finished
0620 23:01:12.722788 (+ 626us) write_op.cc:301] APPLY: starting
0620 23:01:12.722807 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:12.722839 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:12.722841 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:12.722948 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:12.722970 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:12.723062 (+ 92us) write_op.cc:312] APPLY: finished
0620 23:01:12.723242 (+ 180us) log.cc:842] Serialized 22 byte log entry
0620 23:01:12.723337 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:12.723351 (+ 14us) write_op.cc:454] Released schema lock
0620 23:01:12.723358 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:12.723432 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:12.723436 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":280,"threads_started":1,"wal-append.queue_time_us":413,"child_traces":[["op",{"apply.queue_time_us":353,"apply.run_cpu_time_us":534,"apply.run_wall_time_us":528,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":548,"prepare.run_cpu_time_us":341,"prepare.run_wall_time_us":342,"replication_time_us":1400304,"thread_start_us":656,"threads_started":2}],["op",{"apply.queue_time_us":566,"apply.run_cpu_time_us":667,"apply.run_wall_time_us":710,"mutex_wait_us":26,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":342,"prepare.run_cpu_time_us":421,"prepare.run_wall_time_us":423,"replication_time_us":675730,"thread_start_us":614,"threads_started":2}]]}
I20260620 23:01:13.069118 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:13.069329 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:13.070179 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:13.070858 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:13.070997 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:13.071736 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:13.075917 13699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=15, attempt_no=0}) took 1751 ms. Trace:
I20260620 23:01:13.076016 13699 rpcz_store.cc:276] 0620 23:01:11.324421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:11.324492 (+ 71us) service_pool.cc:227] Handling call
0620 23:01:13.075856 (+1751364us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:11.640415 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:11.640538 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:11.640544 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:11.640547 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:11.640663 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:11.640674 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:11.640679 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:11.640734 (+ 55us) tablet.cc:699] Row locks acquired
0620 23:01:11.640737 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:11.640778 (+ 41us) write_op.cc:270] Start()
0620 23:01:11.764937 (+124159us) write_op.cc:276] Timestamp: P: 1781996471640779 usec, L: 0
0620 23:01:11.764942 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:11.765171 (+ 229us) log.cc:842] Serialized 354 byte log entry
0620 23:01:13.074942 (+1309771us) op_driver.cc:444] REPLICATION: finished
0620 23:01:13.075395 (+ 453us) write_op.cc:301] APPLY: starting
0620 23:01:13.075409 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:13.075439 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:13.075457 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:13.075513 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:13.075536 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:13.075540 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:13.075636 (+ 96us) log.cc:842] Serialized 22 byte log entry
0620 23:01:13.075744 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:13.075761 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:13.075767 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":401,"num_ops":1,"prepare.queue_time_us":315507,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":124862,"replication_time_us":1309974,"thread_start_us":297,"threads_started":1}]]}
I20260620 23:01:13.471112 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:13.492127 13699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=16, attempt_no=0}) took 1131 ms. Trace:
I20260620 23:01:13.492322 13699 rpcz_store.cc:276] 0620 23:01:12.360351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:12.360415 (+ 64us) service_pool.cc:227] Handling call
0620 23:01:13.492029 (+1131614us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:12.361124 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:12.361274 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:12.361279 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:12.361281 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:12.361379 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:12.361390 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:12.361394 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:12.361443 (+ 49us) tablet.cc:699] Row locks acquired
0620 23:01:12.361445 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:12.361475 (+ 30us) write_op.cc:270] Start()
0620 23:01:12.624695 (+263220us) write_op.cc:276] Timestamp: P: 1781996472361536 usec, L: 0
0620 23:01:12.624700 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:12.625012 (+ 312us) log.cc:842] Serialized 354 byte log entry
0620 23:01:13.491305 (+866293us) op_driver.cc:444] REPLICATION: finished
0620 23:01:13.491484 (+ 179us) write_op.cc:301] APPLY: starting
0620 23:01:13.491504 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:13.491549 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:13.491576 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:13.491655 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:13.491686 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:13.491691 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:13.491785 (+ 94us) log.cc:842] Serialized 22 byte log entry
0620 23:01:13.491899 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:13.491919 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:13.491926 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"num_ops":1,"prepare.queue_time_us":328,"prepare.run_cpu_time_us":920,"prepare.run_wall_time_us":263993,"replication_time_us":866571,"thread_start_us":235,"threads_started":1}]]}
W20260620 23:01:13.494638 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.984s user 0.001s sys 0.000s
W20260620 23:01:13.494690 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.984s user 0.001s sys 0.000s
I20260620 23:01:14.295749 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:14.295938 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:14.296417 13391 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.129:55473 (request call id 1110) took 1571 ms. Trace:
I20260620 23:01:14.296813 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:14.297282 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:14.297369 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:14.296587 13391 rpcz_store.cc:276] 0620 23:01:12.725246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:12.725349 (+ 103us) service_pool.cc:227] Handling call
0620 23:01:12.725553 (+ 204us) raft_consensus.cc:1457] Updating replica for 3 ops
0620 23:01:12.725669 (+ 116us) raft_consensus.cc:1520] Early marking committed up to index 0
0620 23:01:12.725672 (+ 3us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0620 23:01:12.725756 (+ 84us) write_op.cc:270] Start()
0620 23:01:13.364918 (+639162us) write_op.cc:276] Timestamp: P: 1781996471004824 usec, L: 0
0620 23:01:13.365444 (+ 526us) write_op.cc:270] Start()
0620 23:01:13.880651 (+515207us) write_op.cc:276] Timestamp: P: 1781996471640779 usec, L: 0
0620 23:01:13.881232 (+ 581us) write_op.cc:270] Start()
0620 23:01:14.294419 (+413187us) write_op.cc:276] Timestamp: P: 1781996472361536 usec, L: 0
0620 23:01:14.294806 (+ 387us) log.cc:842] Serialized 1062 byte log entry
0620 23:01:14.295245 (+ 439us) raft_consensus.cc:1631] Marking committed up to 16
0620 23:01:14.295623 (+ 378us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:14.295661 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:14.296268 (+ 607us) raft_consensus.cc:1671] finished
0620 23:01:14.296271 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:14.296357 (+ 86us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:13.365402 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:13.365507 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:13.365512 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:13.365513 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:13.365593 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:13.365606 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:13.365609 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:13.365654 (+ 45us) tablet.cc:699] Row locks acquired
0620 23:01:13.365655 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:14.295278 (+929623us) op_driver.cc:444] REPLICATION: finished
0620 23:01:14.295692 (+ 414us) write_op.cc:301] APPLY: starting
0620 23:01:14.295711 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:14.295759 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:14.295761 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:14.295999 (+ 238us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:14.296034 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:14.296061 (+ 27us) write_op.cc:312] APPLY: finished
0620 23:01:14.296140 (+ 79us) log.cc:842] Serialized 22 byte log entry
0620 23:01:14.296217 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:14.296238 (+ 21us) write_op.cc:454] Released schema lock
0620 23:01:14.296248 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:14.296338 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:14.296342 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:13.881162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:13.881277 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:13.881283 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:13.881284 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:13.881369 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:13.881382 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:13.881385 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:13.881449 (+ 64us) tablet.cc:699] Row locks acquired
0620 23:01:13.881451 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:01:14.294646 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:14.294754 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:14.294760 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:14.294761 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:14.294839 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:14.294850 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:14.294854 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:14.294927 (+ 73us) tablet.cc:699] Row locks acquired
0620 23:01:14.294931 (+ 4us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":359,"threads_started":1,"wal-append.queue_time_us":437,"child_traces":[["op",{"apply.queue_time_us":361,"apply.run_cpu_time_us":705,"apply.run_wall_time_us":704,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":350,"prepare.run_cpu_time_us":336,"prepare.run_wall_time_us":337,"replication_time_us":1569509,"thread_start_us":512,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":369,"prepare.run_cpu_time_us":372,"prepare.run_wall_time_us":372,"thread_start_us":230,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":366,"prepare.run_wall_time_us":366}]]}
I20260620 23:01:14.298017 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:14.298310 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:14.365845 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:14.366084 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:14.366990 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:14.367461 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:14.367623 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:14.368397 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:14.412683 13709 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=17, attempt_no=0}) took 1335 ms. Trace:
I20260620 23:01:14.412871 13709 rpcz_store.cc:276] 0620 23:01:13.077630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:13.077722 (+ 92us) service_pool.cc:227] Handling call
0620 23:01:14.412606 (+1334884us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:13.078285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:13.078374 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:13.078378 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:13.078379 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:13.078518 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:13.078530 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:13.078537 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:13.078588 (+ 51us) tablet.cc:699] Row locks acquired
0620 23:01:13.078589 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:13.078620 (+ 31us) write_op.cc:270] Start()
0620 23:01:13.710761 (+632141us) write_op.cc:276] Timestamp: P: 1781996473078636 usec, L: 0
0620 23:01:13.710765 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:13.711089 (+ 324us) log.cc:842] Serialized 354 byte log entry
0620 23:01:14.410438 (+699349us) op_driver.cc:444] REPLICATION: finished
0620 23:01:14.410975 (+ 537us) write_op.cc:301] APPLY: starting
0620 23:01:14.410993 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:14.411029 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:14.411075 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:14.411166 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:14.411194 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:14.411198 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:14.411282 (+ 84us) log.cc:842] Serialized 22 byte log entry
0620 23:01:14.412420 (+ 1138us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:14.412442 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:14.412449 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":469,"num_ops":1,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":825,"prepare.run_wall_time_us":632896,"replication_time_us":699634,"thread_start_us":311,"threads_started":1}]]}
W20260620 23:01:14.414371 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.769s user 0.001s sys 0.000s
W20260620 23:01:14.414420 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.769s user 0.001s sys 0.000s
I20260620 23:01:14.614562 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:14.623335 13709 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=18, attempt_no=0}) took 1128 ms. Trace:
I20260620 23:01:14.623508 13709 rpcz_store.cc:276] 0620 23:01:13.495069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:13.495147 (+ 78us) service_pool.cc:227] Handling call
0620 23:01:14.623248 (+1128101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:13.711249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:13.711375 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:13.711381 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:13.711383 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:13.711477 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:13.711490 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:13.711497 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:13.711556 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:01:13.711559 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:13.711594 (+ 35us) write_op.cc:270] Start()
0620 23:01:13.917762 (+206168us) write_op.cc:276] Timestamp: P: 1781996473711618 usec, L: 0
0620 23:01:13.917768 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:13.918035 (+ 267us) log.cc:842] Serialized 354 byte log entry
0620 23:01:14.622681 (+704646us) op_driver.cc:444] REPLICATION: finished
0620 23:01:14.622778 (+ 97us) write_op.cc:301] APPLY: starting
0620 23:01:14.622802 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:14.622838 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:14.622866 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:14.622920 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:14.622946 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:14.622951 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:14.623044 (+ 93us) log.cc:842] Serialized 22 byte log entry
0620 23:01:14.623124 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:14.623149 (+ 25us) write_op.cc:454] Released schema lock
0620 23:01:14.623155 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"num_ops":1,"prepare.queue_time_us":215612,"prepare.run_cpu_time_us":828,"prepare.run_wall_time_us":206898,"replication_time_us":704887}]]}
I20260620 23:01:14.661762 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:14.662155 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:14.662917 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:14.663473 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:14.663967 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:14.664516 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:14.664752 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:15.167460 13592 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:15.167667 13592 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.168776 13592 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.169332 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:15.169368 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.170071 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:15.170317 13592 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260620 23:01:15.184623 13616 scanner-internal.cc:458] Time spent opening tablet: real 0.620s user 0.001s sys 0.000s
W20260620 23:01:15.184937 13615 scanner-internal.cc:458] Time spent opening tablet: real 0.620s user 0.001s sys 0.000s
I20260620 23:01:15.203658 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:15.203843 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.204600 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.204914 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:15.205211 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.205761 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:15.631742 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:15.701032 13716 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:15.701220 13716 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.701844 13716 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.702203 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:15.706405 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.707077 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:15.707345 13716 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:15.747092 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:15.747270 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.747411 13716 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:15.747541 13716 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.747874 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.748175 13716 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.748497 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:15.748553 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.749183 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:15.749204 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.749455 13521 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 1.
I20260620 23:01:15.749879 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:15.750155 13716 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:15.750365 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5; no voters: 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:15.750625 13626 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:01:15.750731 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6dce4229f6f247c7b2b68b3a75c39539)
I20260620 23:01:15.750820 13626 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:15.753620 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.754179 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.754736 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 election: Requested vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.754806 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539"
I20260620 23:01:15.755414 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5"
I20260620 23:01:15.755934 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:15.756246 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:01:15.755752 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260620 23:01:15.758693 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.759212 13704 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.760862 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.761329 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.762938 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:15.763128 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.763460 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.763758 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.764075 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:15.764427 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.764655 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.764904 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:15.765172 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 23:01:15.766381 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.766873 13704 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.768694 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.769143 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.770902 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.771412 13704 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.773133 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.773587 13704 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.773837 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:15.774014 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.774549 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.774835 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:15.775020 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.775399 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.775609 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45; no voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:15.775915 13626 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260620 23:01:15.776024 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.777587 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.778008 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.779767 13391 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Rejecting Update request from peer 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:15.780063 13717 raft_consensus.cc:3032] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:01:15.780206 13717 raft_consensus.cc:729] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6dce4229f6f247c7b2b68b3a75c39539, State: Running, Role: LEADER
I20260620 23:01:15.780243 13707 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 }, Status: INVALID_TERM, Last received: 1.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.780723 13717 consensus_queue.cc:260] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 21, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.781313 13717 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:15.784590 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:15.784720 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.785223 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.785567 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:15.785738 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:15.785830 13704 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:15.785982 13704 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.786339 13262 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 2.
I20260620 23:01:15.786672 13704 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:15.786742 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539; no voters: eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:15.786981 13626 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:01:15.786984 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:15.787084 13626 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:15.787192 13521 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dce4229f6f247c7b2b68b3a75c39539 in term 1.
I20260620 23:01:15.787207 13626 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:01:15.787613 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:15.787982 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5; no voters:
I20260620 23:01:15.788209 13704 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:01:15.788335 13704 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:15.788420 13704 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:01:15.789526 13626 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.789894 13391 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 in current term 3: Already voted for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in this term.
I20260620 23:01:15.790319 13626 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 election: Requested vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:15.790560 13704 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.790706 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5"
I20260620 23:01:15.790707 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539"
I20260620 23:01:15.790881 13521 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 1 FOLLOWER]: Advancing to term 3
I20260620 23:01:15.790951 13262 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in current term 3: Already voted for candidate 6dce4229f6f247c7b2b68b3a75c39539 in this term.
I20260620 23:01:15.791224 13704 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 3 election: Requested vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:15.791800 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
I20260620 23:01:15.792048 13391 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 in current term 3: Already voted for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in this term.
I20260620 23:01:15.792928 13521 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 3.
I20260620 23:01:15.793267 13325 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5; no voters: 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:15.793495 13626 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:01:15.793601 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 3 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5"
I20260620 23:01:15.793870 13521 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 in current term 3: Already voted for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in this term.
I20260620 23:01:15.793982 13626 raft_consensus.cc:686] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 LEADER]: Becoming Leader. State: Replica: 6a46fb3b701f43cb96ea5c4958cb0a45, State: Running, Role: LEADER
I20260620 23:01:15.794287 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:15.794551 13704 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 23:01:15.794471 13626 consensus_queue.cc:237] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 21, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:15.797590 13262 raft_consensus.cc:1261] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Refusing update from remote peer 6a46fb3b701f43cb96ea5c4958cb0a45: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
I20260620 23:01:15.797435 13120 catalog_manager.cc:5495] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 reported cstate change: term changed from 1 to 3, leader changed from 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129) to 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130). New cstate: current_term: 3 leader_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:15.798202 13521 raft_consensus.cc:1261] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Refusing update from remote peer 6a46fb3b701f43cb96ea5c4958cb0a45: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
I20260620 23:01:15.798257 13626 consensus_queue.cc:1035] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260620 23:01:15.798691 13718 consensus_queue.cc:1035] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260620 23:01:16.090513 13691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49000: Illegal state: Replica 6dce4229f6f247c7b2b68b3a75c39539 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } } }
I20260620 23:01:16.091087 13691 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:49000 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=20, attempt_no=0}) took 1465 ms. Trace:
I20260620 23:01:16.091221 13691 rpcz_store.cc:276] 0620 23:01:14.625340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:14.625391 (+ 51us) service_pool.cc:227] Handling call
0620 23:01:16.091020 (+1465629us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:15.104289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:15.104417 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:15.104423 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:15.104426 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:15.104520 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:15.104531 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:15.104536 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:15.104592 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:01:15.104594 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:15.104638 (+ 44us) write_op.cc:270] Start()
0620 23:01:16.089791 (+985153us) write_op.cc:276] Timestamp: P: 1781996475104617 usec, L: 0
0620 23:01:16.089797 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:16.090373 (+ 576us) op_driver.cc:372] HandleFailure(Illegal state: Replica 6dce4229f6f247c7b2b68b3a75c39539 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } } })
0620 23:01:16.090410 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:16.090428 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:16.090435 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":478367}]]}
W20260620 23:01:16.092108 13691 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49000: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.002s ago.
W20260620 23:01:16.108603 13481 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43354: Illegal state: replica eb45f804a3ff4b12a653ee059501b6f5 is not leader of this config: current role FOLLOWER
W20260620 23:01:16.108685 13482 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43354: Illegal state: replica eb45f804a3ff4b12a653ee059501b6f5 is not leader of this config: current role FOLLOWER
I20260620 23:01:16.144693 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:16.144910 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:16.145637 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:16.146004 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:16.146040 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:16.146725 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:16.146963 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:16.272832 13222 tablet_service.cc:2970] error setting up scanner ed6670118bea4ea3af51613360220b03 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.152s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:16.272872 13220 tablet_service.cc:2970] error setting up scanner 3f66809f4d5e4753942f32de72e1387f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.152s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
W20260620 23:01:16.286900 13220 tablet_service.cc:2970] error setting up scanner cae47f3a8e234aea81fed0ad2956f624 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.166s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
W20260620 23:01:16.294116 13220 tablet_service.cc:2970] error setting up scanner b0bb542ddd874fe3a1f2198857ba4ac3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.173s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:16.317018 13220 tablet_service.cc:2970] error setting up scanner 38ba04d8a6684a94a8940a6e46a7edba with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.196s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:16.323356 13220 tablet_service.cc:2970] error setting up scanner b1072ccf942249a48f20316d19229663 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.203s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
W20260620 23:01:16.372001 13220 tablet_service.cc:2970] error setting up scanner dcf0513e5ee24b6ca01e8d44c0753bcf with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.251s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:16.377872 13220 tablet_service.cc:2970] error setting up scanner 46dc591c6ead4acc84411765e344cd9a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.257s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
W20260620 23:01:16.476718 13220 tablet_service.cc:2970] error setting up scanner 9c808742254e43ac8e2f6e044820fbbb with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.356s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
W20260620 23:01:16.494899 13220 tablet_service.cc:2970] error setting up scanner 2983d0ec97c94e35b651a315e61443e3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.374s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
I20260620 23:01:16.535634 13716 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:16.535840 13716 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:16.536715 13716 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:16.537302 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 23 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:16.537293 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 23 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:16.538206 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:16.538508 13716 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:16.737974 13220 tablet_service.cc:2970] error setting up scanner 81ce7211209a4cc98d4114d0c8ebcb37 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.617s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:16.767629 13220 tablet_service.cc:2970] error setting up scanner 15da8bf3a9a74450b660e4a801e8a998 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.647s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:17.095686 13716 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:17.095973 13716 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:17.096781 13716 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:17.097220 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:17.097295 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:17.097999 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:17.098531 13716 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:17.104348 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:17.104521 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:17.105787 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:17.106277 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 23 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:17.106495 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 23 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:17.107014 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:17.107430 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:17.282449 13220 tablet_service.cc:2970] error setting up scanner 6519600d50ca4cd4b6b2421796eb3f02 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.175s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
W20260620 23:01:17.348847 13220 tablet_service.cc:2970] error setting up scanner 104f800d93fb4b8bbe8c1197449dcde0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.241s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
I20260620 23:01:17.819264 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:17.819466 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:17.820497 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:17.821012 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:17.821161 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:17.821758 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
W20260620 23:01:17.991977 13220 tablet_service.cc:2970] error setting up scanner 155077cc90c548d08c9b53c74e85039f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.173s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:18.053470 13220 tablet_service.cc:2970] error setting up scanner 32035cc47df640ab967bc86fdfe48fcc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.234s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:18.132336 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:18.132643 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:18.133793 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:18.134267 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:18.134620 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:18.135202 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:18.150676 13747 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=22, attempt_no=0}) took 1585 ms. Trace:
I20260620 23:01:18.150820 13747 rpcz_store.cc:276] 0620 23:01:16.564754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:16.564817 (+ 63us) service_pool.cc:227] Handling call
0620 23:01:18.150615 (+1585798us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:16.879235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:16.879354 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:16.879361 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:16.879364 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:16.879462 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:16.879475 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:16.879479 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:16.879538 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:01:16.879540 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:16.879583 (+ 43us) write_op.cc:270] Start()
0620 23:01:17.638840 (+759257us) write_op.cc:276] Timestamp: P: 1781996476879626 usec, L: 0
0620 23:01:17.638848 (+ 8us) op_driver.cc:331] REPLICATION: starting
0620 23:01:17.639095 (+ 247us) log.cc:842] Serialized 354 byte log entry
0620 23:01:18.149342 (+510247us) op_driver.cc:444] REPLICATION: finished
0620 23:01:18.150031 (+ 689us) write_op.cc:301] APPLY: starting
0620 23:01:18.150048 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:18.150085 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:18.150113 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:18.150201 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:18.150228 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:18.150233 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:18.150319 (+ 86us) log.cc:842] Serialized 22 byte log entry
0620 23:01:18.150469 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:18.150489 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:18.150496 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":621,"num_ops":1,"prepare.queue_time_us":314057,"prepare.run_cpu_time_us":847,"prepare.run_wall_time_us":759975,"replication_time_us":510461,"thread_start_us":482,"threads_started":1}]]}
I20260620 23:01:18.291517 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:18.298352 13747 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=23, attempt_no=0}) took 1186 ms. Trace:
I20260620 23:01:18.298511 13747 rpcz_store.cc:276] 0620 23:01:17.111586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:17.111669 (+ 83us) service_pool.cc:227] Handling call
0620 23:01:18.298275 (+1186606us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:17.639262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:17.639405 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:17.639413 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:17.639415 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:17.639530 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:17.639543 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:17.639549 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:17.639620 (+ 71us) tablet.cc:699] Row locks acquired
0620 23:01:17.639623 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:17.639673 (+ 50us) write_op.cc:270] Start()
0620 23:01:18.001857 (+362184us) write_op.cc:276] Timestamp: P: 1781996477639642 usec, L: 0
0620 23:01:18.001862 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:18.002243 (+ 381us) log.cc:842] Serialized 354 byte log entry
0620 23:01:18.297604 (+295361us) op_driver.cc:444] REPLICATION: finished
0620 23:01:18.297741 (+ 137us) write_op.cc:301] APPLY: starting
0620 23:01:18.297761 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:18.297809 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:18.297831 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:18.297902 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:18.297948 (+ 46us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:18.297953 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:18.298031 (+ 78us) log.cc:842] Serialized 22 byte log entry
0620 23:01:18.298139 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:18.298158 (+ 19us) write_op.cc:454] Released schema lock
0620 23:01:18.298165 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":75,"num_ops":1,"prepare.queue_time_us":527163,"prepare.run_cpu_time_us":978,"prepare.run_wall_time_us":363068,"replication_time_us":295710}]]}
I20260620 23:01:18.597359 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:18.662479 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:18.662693 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:18.663595 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:18.664065 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:18.664227 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:18.664780 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:18.819371 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:19.624899 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:19.625146 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:19.626034 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:19.626595 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:19.626664 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:19.627286 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
W20260620 23:01:20.170994 13220 tablet_service.cc:2970] error setting up scanner 245dffa0a9a94b549ec29f8db720b171 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.572s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:20.488906 13220 tablet_service.cc:2970] error setting up scanner 639eccfea53d468cb1f411eec5f202b9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.889s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:20.518173 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:20.524513 13761 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=27, attempt_no=0}) took 1700 ms. Trace:
I20260620 23:01:20.524711 13761 rpcz_store.cc:276] 0620 23:01:18.824190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:18.824282 (+ 92us) service_pool.cc:227] Handling call
0620 23:01:20.524447 (+1700165us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:19.003295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:19.003456 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:19.003462 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:19.003464 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:19.003558 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:19.003571 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:19.003578 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:19.003636 (+ 58us) tablet.cc:699] Row locks acquired
0620 23:01:19.003638 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:19.003685 (+ 47us) write_op.cc:270] Start()
0620 23:01:19.197885 (+194200us) write_op.cc:276] Timestamp: P: 1781996479003726 usec, L: 0
0620 23:01:19.197890 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:19.198176 (+ 286us) log.cc:842] Serialized 354 byte log entry
0620 23:01:20.523620 (+1325444us) op_driver.cc:444] REPLICATION: finished
0620 23:01:20.523984 (+ 364us) write_op.cc:301] APPLY: starting
0620 23:01:20.523998 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:20.524027 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:20.524049 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:20.524129 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:20.524153 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:20.524156 (+ 3us) write_op.cc:312] APPLY: finished
0620 23:01:20.524231 (+ 75us) log.cc:842] Serialized 22 byte log entry
0620 23:01:20.524292 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:20.524304 (+ 12us) write_op.cc:454] Released schema lock
0620 23:01:20.524309 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":312,"num_ops":1,"prepare.queue_time_us":178613,"prepare.run_cpu_time_us":901,"prepare.run_wall_time_us":194978,"replication_time_us":1325704,"thread_start_us":226,"threads_started":1}]]}
I20260620 23:01:20.782182 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:20.782401 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:20.782629 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 209) took 2183 ms. Trace:
I20260620 23:01:20.783334 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:20.782793 13262 rpcz_store.cc:276] 0620 23:01:18.598885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:18.598932 (+ 47us) service_pool.cc:227] Handling call
0620 23:01:18.599141 (+ 209us) raft_consensus.cc:1457] Updating replica for 3 ops
0620 23:01:18.599271 (+ 130us) raft_consensus.cc:1520] Early marking committed up to index 25
0620 23:01:18.599581 (+ 310us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0620 23:01:18.599756 (+ 175us) write_op.cc:270] Start()
0620 23:01:19.315945 (+716189us) write_op.cc:276] Timestamp: P: 1781996477639642 usec, L: 0
0620 23:01:19.316602 (+ 657us) write_op.cc:270] Start()
0620 23:01:19.938793 (+622191us) write_op.cc:276] Timestamp: P: 1781996478153635 usec, L: 0
0620 23:01:19.939381 (+ 588us) write_op.cc:270] Start()
0620 23:01:20.780581 (+841200us) write_op.cc:276] Timestamp: P: 1781996478301532 usec, L: 0
0620 23:01:20.781266 (+ 685us) log.cc:842] Serialized 1062 byte log entry
0620 23:01:20.781654 (+ 388us) raft_consensus.cc:1631] Marking committed up to 26
0620 23:01:20.782055 (+ 401us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:20.782099 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:20.782420 (+ 321us) raft_consensus.cc:1671] finished
0620 23:01:20.782424 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:20.782555 (+ 131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:19.316512 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:19.316619 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:19.316625 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:19.316626 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:19.316724 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:19.316738 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:19.316742 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:19.316787 (+ 45us) tablet.cc:699] Row locks acquired
0620 23:01:19.316789 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:20.781693 (+1464904us) op_driver.cc:444] REPLICATION: finished
0620 23:01:20.782155 (+ 462us) write_op.cc:301] APPLY: starting
0620 23:01:20.782176 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:20.782236 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:20.782238 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:20.782311 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:20.782344 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:20.782423 (+ 79us) write_op.cc:312] APPLY: finished
0620 23:01:20.782539 (+ 116us) log.cc:842] Serialized 22 byte log entry
0620 23:01:20.782621 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:20.782637 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:20.782644 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:20.782724 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:20.782728 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:19.939278 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:19.939398 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:19.939403 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:19.939404 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:19.939482 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:19.939495 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:19.939498 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:19.939543 (+ 45us) tablet.cc:699] Row locks acquired
0620 23:01:19.939545 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:01:20.781109 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:20.781235 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:20.781240 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:20.781241 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:20.781321 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:20.781334 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:20.781337 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:20.781384 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:20.781386 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":299,"threads_started":1,"wal-append.queue_time_us":383,"child_traces":[["op",{"apply.queue_time_us":404,"apply.run_cpu_time_us":641,"apply.run_wall_time_us":641,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":420,"prepare.run_cpu_time_us":375,"prepare.run_wall_time_us":374,"replication_time_us":2181923,"thread_start_us":599,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":351,"prepare.run_cpu_time_us":360,"prepare.run_wall_time_us":360,"thread_start_us":218,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":354,"prepare.run_cpu_time_us":378,"prepare.run_wall_time_us":378,"thread_start_us":216,"threads_started":1}]]}
I20260620 23:01:20.783882 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:20.784132 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:20.784765 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:20.785109 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:21.027841 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:21.028093 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:21.028808 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:21.029276 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:21.029400 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:21.030259 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:21.030870 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:21.068665 13769 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=28, attempt_no=0}) took 1422 ms. Trace:
I20260620 23:01:21.068805 13769 rpcz_store.cc:276] 0620 23:01:19.646087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:19.646175 (+ 88us) service_pool.cc:227] Handling call
0620 23:01:21.068606 (+1422431us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:19.646642 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:19.646730 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:19.646734 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:19.646735 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:19.646803 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:19.646811 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:19.646814 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:19.646862 (+ 48us) tablet.cc:699] Row locks acquired
0620 23:01:19.646863 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:19.646894 (+ 31us) write_op.cc:270] Start()
0620 23:01:20.326078 (+679184us) write_op.cc:276] Timestamp: P: 1781996479646887 usec, L: 0
0620 23:01:20.326084 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:20.326349 (+ 265us) log.cc:842] Serialized 354 byte log entry
0620 23:01:21.067375 (+741026us) op_driver.cc:444] REPLICATION: finished
0620 23:01:21.067923 (+ 548us) write_op.cc:301] APPLY: starting
0620 23:01:21.067945 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:21.067993 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:21.068028 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:21.068095 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:21.068173 (+ 78us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:21.068183 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:01:21.068316 (+ 133us) log.cc:842] Serialized 22 byte log entry
0620 23:01:21.068452 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:21.068474 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:21.068481 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":482,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":697,"prepare.run_wall_time_us":679777,"replication_time_us":741250,"thread_start_us":321,"threads_started":1}]]}
I20260620 23:01:21.441360 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:21.441573 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:21.442476 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:21.442868 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:21.443153 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:21.443799 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:21.444063 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:22.143115 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:22.143304 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:22.144130 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:22.144697 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:22.145663 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:22.146247 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:22.173614 13775 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=30, attempt_no=0}) took 1103 ms. Trace:
I20260620 23:01:22.173811 13775 rpcz_store.cc:276] 0620 23:01:21.070584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:21.070647 (+ 63us) service_pool.cc:227] Handling call
0620 23:01:22.173544 (+1102897us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:21.179886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:21.180037 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:21.180044 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:21.180046 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:21.180155 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:21.180168 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:21.180175 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:21.180239 (+ 64us) tablet.cc:699] Row locks acquired
0620 23:01:21.180241 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:21.180280 (+ 39us) write_op.cc:270] Start()
0620 23:01:21.461414 (+281134us) write_op.cc:276] Timestamp: P: 1781996481180280 usec, L: 0
0620 23:01:21.461418 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:21.461622 (+ 204us) log.cc:842] Serialized 354 byte log entry
0620 23:01:22.172268 (+710646us) op_driver.cc:444] REPLICATION: finished
0620 23:01:22.172854 (+ 586us) write_op.cc:301] APPLY: starting
0620 23:01:22.172874 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:22.172919 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:22.172961 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:22.173083 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:22.173134 (+ 51us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:22.173141 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:22.173288 (+ 147us) log.cc:842] Serialized 22 byte log entry
0620 23:01:22.173403 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:22.173424 (+ 21us) write_op.cc:454] Released schema lock
0620 23:01:22.173432 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":510,"num_ops":1,"prepare.queue_time_us":108855,"prepare.run_cpu_time_us":735,"prepare.run_wall_time_us":281802,"replication_time_us":710809,"thread_start_us":366,"threads_started":1}]]}
I20260620 23:01:22.773659 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:22.773721 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 210) took 1988 ms. Trace:
I20260620 23:01:22.773873 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:22.774875 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:22.773876 13262 rpcz_store.cc:276] 0620 23:01:20.785014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:20.785093 (+ 79us) service_pool.cc:227] Handling call
0620 23:01:20.785347 (+ 254us) raft_consensus.cc:1457] Updating replica for 3 ops
0620 23:01:20.785470 (+ 123us) raft_consensus.cc:1520] Early marking committed up to index 28
0620 23:01:20.785981 (+ 511us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0620 23:01:20.786140 (+ 159us) write_op.cc:270] Start()
0620 23:01:21.469365 (+683225us) write_op.cc:276] Timestamp: P: 1781996478691815 usec, L: 0
0620 23:01:21.470013 (+ 648us) write_op.cc:270] Start()
0620 23:01:22.302192 (+832179us) write_op.cc:276] Timestamp: P: 1781996479003726 usec, L: 0
0620 23:01:22.302808 (+ 616us) write_op.cc:270] Start()
0620 23:01:22.771994 (+469186us) write_op.cc:276] Timestamp: P: 1781996479646887 usec, L: 0
0620 23:01:22.772468 (+ 474us) log.cc:842] Serialized 1062 byte log entry
0620 23:01:22.772805 (+ 337us) raft_consensus.cc:1631] Marking committed up to 30
0620 23:01:22.773518 (+ 713us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:22.773557 (+ 39us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:22.773573 (+ 16us) raft_consensus.cc:1671] finished
0620 23:01:22.773576 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:22.773671 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:21.469887 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:21.470065 (+ 178us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:21.470069 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:21.470071 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:21.470155 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:21.470167 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:21.470170 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:21.470213 (+ 43us) tablet.cc:699] Row locks acquired
0620 23:01:21.470214 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:22.772837 (+1302623us) op_driver.cc:444] REPLICATION: finished
0620 23:01:22.773258 (+ 421us) write_op.cc:301] APPLY: starting
0620 23:01:22.773276 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:22.773328 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:22.773333 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:22.773464 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:22.773507 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:22.773544 (+ 37us) write_op.cc:312] APPLY: finished
0620 23:01:22.773652 (+ 108us) log.cc:842] Serialized 22 byte log entry
0620 23:01:22.773724 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:22.773741 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:22.773750 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:22.773831 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:22.773836 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:22.302761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:22.302873 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:22.302879 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:22.302880 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:22.302959 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:22.302971 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:22.302974 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:22.303028 (+ 54us) tablet.cc:699] Row locks acquired
0620 23:01:22.303029 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:22.773192 (+470163us) op_driver.cc:444] REPLICATION: finished
0620 23:01:22.773631 (+ 439us) write_op.cc:301] APPLY: starting
0620 23:01:22.773652 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:22.773841 (+ 189us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:22.773845 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:22.774004 (+ 159us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:22.774036 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:22.774069 (+ 33us) write_op.cc:312] APPLY: finished
0620 23:01:22.774172 (+ 103us) log.cc:842] Serialized 22 byte log entry
Related trace 'op':
0620 23:01:22.772240 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:22.772352 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:22.772358 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:22.772360 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:22.772437 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:22.772449 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:22.772452 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:22.772498 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:01:22.772499 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":253,"threads_started":1,"wal-append.queue_time_us":341,"child_traces":[["op",{"apply.queue_time_us":374,"apply.run_cpu_time_us":646,"apply.run_wall_time_us":646,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":392,"prepare.run_cpu_time_us":448,"prepare.run_wall_time_us":447,"replication_time_us":1986687,"thread_start_us":585,"threads_started":2}],["op",{"apply.queue_time_us":387,"apply.run_cpu_time_us":596,"apply.run_wall_time_us":886,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":408,"prepare.run_cpu_time_us":353,"prepare.run_wall_time_us":353,"replication_time_us":1303166,"thread_start_us":556,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":325,"prepare.run_wall_time_us":326}]]}
I20260620 23:01:22.775399 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:22.775664 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:22.776192 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:22.776484 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:22.827610 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:22.874244 13784 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=31, attempt_no=0}) took 1424 ms. Trace:
I20260620 23:01:22.874370 13784 rpcz_store.cc:276] 0620 23:01:21.449694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:21.449797 (+ 103us) service_pool.cc:227] Handling call
0620 23:01:22.874173 (+1424376us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:21.461735 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:21.461866 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:21.461871 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:21.461873 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:21.461988 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:21.461998 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:21.462002 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:21.462061 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:01:21.462064 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:21.462112 (+ 48us) write_op.cc:270] Start()
0620 23:01:21.812340 (+350228us) write_op.cc:276] Timestamp: P: 1781996481462144 usec, L: 0
0620 23:01:21.812347 (+ 7us) op_driver.cc:331] REPLICATION: starting
0620 23:01:21.812620 (+ 273us) log.cc:842] Serialized 354 byte log entry
0620 23:01:22.873082 (+1060462us) op_driver.cc:444] REPLICATION: finished
0620 23:01:22.873569 (+ 487us) write_op.cc:301] APPLY: starting
0620 23:01:22.873587 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:22.873622 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:22.873670 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:22.873746 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:22.873773 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:22.873777 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:22.873879 (+ 102us) log.cc:842] Serialized 22 byte log entry
0620 23:01:22.874016 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:22.874038 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:22.874045 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":417,"num_ops":1,"prepare.queue_time_us":11474,"prepare.run_cpu_time_us":906,"prepare.run_wall_time_us":350981,"replication_time_us":1060696,"thread_start_us":309,"threads_started":1}]]}
I20260620 23:01:23.499408 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:23.499639 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:23.500489 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:23.501072 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:23.501425 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:23.502216 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:23.514753 13789 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=32, attempt_no=0}) took 1338 ms. Trace:
I20260620 23:01:23.514878 13789 rpcz_store.cc:276] 0620 23:01:22.175827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:22.175903 (+ 76us) service_pool.cc:227] Handling call
0620 23:01:23.514697 (+1338794us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:22.176379 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:22.176473 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:22.176477 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:22.176479 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:22.176550 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:22.176561 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:22.176564 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:22.176604 (+ 40us) tablet.cc:699] Row locks acquired
0620 23:01:22.176606 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:22.176637 (+ 31us) write_op.cc:270] Start()
0620 23:01:22.504811 (+328174us) write_op.cc:276] Timestamp: P: 1781996482176615 usec, L: 0
0620 23:01:22.504816 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:22.505080 (+ 264us) log.cc:842] Serialized 354 byte log entry
0620 23:01:23.513577 (+1008497us) op_driver.cc:444] REPLICATION: finished
0620 23:01:23.514102 (+ 525us) write_op.cc:301] APPLY: starting
0620 23:01:23.514118 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:23.514198 (+ 80us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:23.514226 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:23.514317 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:23.514352 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:23.514358 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:23.514468 (+ 110us) log.cc:842] Serialized 22 byte log entry
0620 23:01:23.514569 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:23.514588 (+ 19us) write_op.cc:454] Released schema lock
0620 23:01:23.514595 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":468,"num_ops":1,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":701,"prepare.run_wall_time_us":328776,"replication_time_us":1008732,"thread_start_us":373,"threads_started":1}]]}
I20260620 23:01:24.134099 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:24.148653 13795 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=33, attempt_no=0}) took 1272 ms. Trace:
I20260620 23:01:24.148790 13795 rpcz_store.cc:276] 0620 23:01:22.876215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:22.876295 (+ 80us) service_pool.cc:227] Handling call
0620 23:01:24.148584 (+1272289us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:22.876872 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:22.876971 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:22.876975 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:22.876977 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:22.877056 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:22.877065 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:22.877068 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:22.877110 (+ 42us) tablet.cc:699] Row locks acquired
0620 23:01:22.877112 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:22.877144 (+ 32us) write_op.cc:270] Start()
0620 23:01:23.421309 (+544165us) write_op.cc:276] Timestamp: P: 1781996482877163 usec, L: 0
0620 23:01:23.421315 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:23.421541 (+ 226us) log.cc:842] Serialized 354 byte log entry
0620 23:01:24.147614 (+726073us) op_driver.cc:444] REPLICATION: finished
0620 23:01:24.148031 (+ 417us) write_op.cc:301] APPLY: starting
0620 23:01:24.148050 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:24.148090 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:24.148119 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:24.148272 (+ 153us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:24.148306 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:24.148313 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:24.148394 (+ 81us) log.cc:842] Serialized 22 byte log entry
0620 23:01:24.148465 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:24.148482 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:24.148488 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":350,"num_ops":1,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":544780,"replication_time_us":726272,"thread_start_us":252,"threads_started":1}]]}
I20260620 23:01:24.507781 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:24.507966 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:24.508776 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:24.509198 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:24.509416 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:24.510064 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
W20260620 23:01:24.525849 13220 tablet_service.cc:2970] error setting up scanner 84517b491fe448c9908cfea267bbc893 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.748s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
W20260620 23:01:24.588092 13220 tablet_service.cc:2970] error setting up scanner b2fd8ce65813481ab40dcf28aad83ecf with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.810s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:25.239126 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 211) took 2462 ms. Trace:
I20260620 23:01:25.239169 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:25.239466 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:25.239324 13262 rpcz_store.cc:276] 0620 23:01:22.777086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:22.777185 (+ 99us) service_pool.cc:227] Handling call
0620 23:01:22.777441 (+ 256us) raft_consensus.cc:1457] Updating replica for 4 ops
0620 23:01:22.777558 (+ 117us) raft_consensus.cc:1520] Early marking committed up to index 31
0620 23:01:22.777639 (+ 81us) raft_consensus.cc:1525] Triggering prepare for 4 ops
0620 23:01:22.777730 (+ 91us) write_op.cc:270] Start()
0620 23:01:23.098919 (+321189us) write_op.cc:276] Timestamp: P: 1781996480527271 usec, L: 0
0620 23:01:23.099346 (+ 427us) write_op.cc:270] Start()
0620 23:01:23.723530 (+624184us) write_op.cc:276] Timestamp: P: 1781996481180280 usec, L: 0
0620 23:01:23.724042 (+ 512us) write_op.cc:270] Start()
0620 23:01:24.306210 (+582168us) write_op.cc:276] Timestamp: P: 1781996481462144 usec, L: 0
0620 23:01:24.306775 (+ 565us) write_op.cc:270] Start()
0620 23:01:25.236948 (+930173us) write_op.cc:276] Timestamp: P: 1781996482176615 usec, L: 0
0620 23:01:25.237631 (+ 683us) log.cc:842] Serialized 1416 byte log entry
0620 23:01:25.238051 (+ 420us) raft_consensus.cc:1631] Marking committed up to 33
0620 23:01:25.238922 (+ 871us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:25.238954 (+ 32us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:25.239000 (+ 46us) raft_consensus.cc:1671] finished
0620 23:01:25.239003 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:25.239063 (+ 60us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:23.099148 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:23.099338 (+ 190us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:23.099344 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:23.099346 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:23.099453 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:23.099468 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:23.099471 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:23.099518 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:23.099519 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:25.238097 (+2138578us) op_driver.cc:444] REPLICATION: finished
0620 23:01:25.238443 (+ 346us) write_op.cc:301] APPLY: starting
0620 23:01:25.238457 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:25.238489 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:25.238490 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:25.238608 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:25.238656 (+ 48us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:25.238786 (+ 130us) write_op.cc:312] APPLY: finished
0620 23:01:25.238911 (+ 125us) log.cc:842] Serialized 22 byte log entry
0620 23:01:25.238977 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:25.238999 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:25.239005 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:25.239126 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:25.239130 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:23.723996 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:23.724101 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:23.724107 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:23.724108 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:23.724192 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:23.724203 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:23.724206 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:23.724253 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:23.724254 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:25.238401 (+1514147us) op_driver.cc:444] REPLICATION: finished
0620 23:01:25.238835 (+ 434us) write_op.cc:301] APPLY: starting
0620 23:01:25.238847 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:25.238880 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:25.238882 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:25.238929 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:25.238966 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:25.239011 (+ 45us) write_op.cc:312] APPLY: finished
0620 23:01:25.239106 (+ 95us) log.cc:842] Serialized 22 byte log entry
0620 23:01:25.239171 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:25.239182 (+ 11us) write_op.cc:454] Released schema lock
0620 23:01:25.239186 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:25.239238 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:25.239242 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:24.306721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:24.306856 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:24.306860 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:24.306862 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:24.306948 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:24.306959 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:24.306962 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:24.307008 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:01:24.307009 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:01:25.237442 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:25.237664 (+ 222us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:25.237669 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:25.237671 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:25.237752 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:25.237764 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:25.237767 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:25.237839 (+ 72us) tablet.cc:699] Row locks acquired
0620 23:01:25.237851 (+ 12us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":380160,"thread_start_us":317,"threads_started":1,"wal-append.queue_time_us":441,"child_traces":[["op",{"apply.queue_time_us":296,"apply.run_cpu_time_us":677,"apply.run_wall_time_us":748,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":459,"prepare.run_wall_time_us":459,"replication_time_us":2460350,"thread_start_us":218,"threads_started":1}],["op",{"apply.queue_time_us":386,"apply.run_cpu_time_us":449,"apply.run_wall_time_us":449,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":322,"prepare.run_cpu_time_us":354,"prepare.run_wall_time_us":354,"replication_time_us":2139043,"thread_start_us":494,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":355,"prepare.run_cpu_time_us":352,"prepare.run_wall_time_us":352,"thread_start_us":236,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":365,"prepare.run_cpu_time_us":472,"prepare.run_wall_time_us":472,"thread_start_us":241,"threads_started":1}]]}
I20260620 23:01:25.240278 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:25.240841 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:25.240846 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:25.241498 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:25.241858 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:25.408495 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:25.422086 13805 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=35, attempt_no=0}) took 1271 ms. Trace:
I20260620 23:01:25.422217 13805 rpcz_store.cc:276] 0620 23:01:24.150871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:24.150958 (+ 87us) service_pool.cc:227] Handling call
0620 23:01:25.422014 (+1271056us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:24.151480 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:24.151571 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:24.151575 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:24.151576 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:24.151653 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:24.151661 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:24.151663 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:24.151782 (+ 119us) tablet.cc:699] Row locks acquired
0620 23:01:24.151784 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:24.151822 (+ 38us) write_op.cc:270] Start()
0620 23:01:24.201001 (+ 49179us) write_op.cc:276] Timestamp: P: 1781996484151797 usec, L: 0
0620 23:01:24.201010 (+ 9us) op_driver.cc:331] REPLICATION: starting
0620 23:01:24.201346 (+ 336us) log.cc:842] Serialized 354 byte log entry
0620 23:01:25.421045 (+1219699us) op_driver.cc:444] REPLICATION: finished
0620 23:01:25.421450 (+ 405us) write_op.cc:301] APPLY: starting
0620 23:01:25.421466 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:25.421500 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:25.421547 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:25.421625 (+ 78us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:25.421651 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:25.421655 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:25.421730 (+ 75us) log.cc:842] Serialized 22 byte log entry
0620 23:01:25.421855 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:25.421875 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:25.421883 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":912,"prepare.run_wall_time_us":49966,"replication_time_us":1220002,"thread_start_us":251,"threads_started":1}]]}
I20260620 23:01:25.646111 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:25.646332 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:25.647228 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:25.647665 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:25.647789 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:25.648334 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:25.648633 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:25.657246 13805 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=36, attempt_no=0}) took 1138 ms. Trace:
I20260620 23:01:25.657426 13805 rpcz_store.cc:276] 0620 23:01:24.518716 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:24.518804 (+ 88us) service_pool.cc:227] Handling call
0620 23:01:25.657139 (+1138335us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:24.519251 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:24.519346 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:24.519350 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:24.519352 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:24.519430 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:24.519438 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:24.519441 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:24.519482 (+ 41us) tablet.cc:699] Row locks acquired
0620 23:01:24.519483 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:24.519514 (+ 31us) write_op.cc:270] Start()
0620 23:01:25.035686 (+516172us) write_op.cc:276] Timestamp: P: 1781996484519529 usec, L: 0
0620 23:01:25.035691 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:25.035930 (+ 239us) log.cc:842] Serialized 354 byte log entry
0620 23:01:25.656429 (+620499us) op_driver.cc:444] REPLICATION: finished
0620 23:01:25.656545 (+ 116us) write_op.cc:301] APPLY: starting
0620 23:01:25.656570 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:25.656610 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:25.656638 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:25.656752 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:25.656795 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:25.656802 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:25.656908 (+ 106us) log.cc:842] Serialized 22 byte log entry
0620 23:01:25.657001 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:25.657022 (+ 21us) write_op.cc:454] Released schema lock
0620 23:01:25.657030 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":710,"prepare.run_wall_time_us":516787,"replication_time_us":620703}]]}
I20260620 23:01:26.265413 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:26.265628 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:26.266409 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:26.266848 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:26.267956 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:26.268507 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:26.268778 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:27.151109 13220 tablet_service.cc:2970] error setting up scanner cf52e8f895714939a8534880e91dfd0a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.908s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:27.353529 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:27.353763 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:27.354594 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:27.355062 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:27.355134 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:27.355798 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:27.356034 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:27.377204 13817 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=38, attempt_no=0}) took 1717 ms. Trace:
I20260620 23:01:27.377363 13817 rpcz_store.cc:276] 0620 23:01:25.659210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:25.659285 (+ 75us) service_pool.cc:227] Handling call
0620 23:01:27.377143 (+1717858us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:26.136415 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:26.136573 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:26.136579 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:26.136582 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:26.136681 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:26.136693 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:26.136700 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:26.136791 (+ 91us) tablet.cc:699] Row locks acquired
0620 23:01:26.136793 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:26.136850 (+ 57us) write_op.cc:270] Start()
0620 23:01:26.629999 (+493149us) write_op.cc:276] Timestamp: P: 1781996486136849 usec, L: 0
0620 23:01:26.630004 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:26.630270 (+ 266us) log.cc:842] Serialized 354 byte log entry
0620 23:01:27.376035 (+745765us) op_driver.cc:444] REPLICATION: finished
0620 23:01:27.376525 (+ 490us) write_op.cc:301] APPLY: starting
0620 23:01:27.376543 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:27.376577 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:27.376600 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:27.376754 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:27.376792 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:27.376796 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:27.376897 (+ 101us) log.cc:842] Serialized 22 byte log entry
0620 23:01:27.376966 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:27.376982 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:27.376989 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":426,"num_ops":1,"prepare.queue_time_us":476690,"prepare.run_cpu_time_us":915,"prepare.run_wall_time_us":493990,"raft.queue_time_us":185,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":745994,"thread_start_us":285,"threads_started":1}]]}
I20260620 23:01:27.581533 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:27.581574 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 212) took 2339 ms. Trace:
I20260620 23:01:27.581785 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:27.582676 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:27.583252 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:27.583267 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:27.581875 13262 rpcz_store.cc:276] 0620 23:01:25.242535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:25.242606 (+ 71us) service_pool.cc:227] Handling call
0620 23:01:25.242804 (+ 198us) raft_consensus.cc:1457] Updating replica for 4 ops
0620 23:01:25.242943 (+ 139us) raft_consensus.cc:1520] Early marking committed up to index 35
0620 23:01:25.243129 (+ 186us) raft_consensus.cc:1525] Triggering prepare for 4 ops
0620 23:01:25.243269 (+ 140us) write_op.cc:270] Start()
0620 23:01:26.216433 (+973164us) write_op.cc:276] Timestamp: P: 1781996482877163 usec, L: 0
0620 23:01:26.217035 (+ 602us) write_op.cc:270] Start()
0620 23:01:26.767206 (+550171us) write_op.cc:276] Timestamp: P: 1781996483517486 usec, L: 0
0620 23:01:26.767808 (+ 602us) write_op.cc:270] Start()
0620 23:01:26.990061 (+222253us) write_op.cc:276] Timestamp: P: 1781996484151797 usec, L: 0
0620 23:01:26.990483 (+ 422us) write_op.cc:270] Start()
0620 23:01:27.578688 (+588205us) write_op.cc:276] Timestamp: P: 1781996484519529 usec, L: 0
0620 23:01:27.579760 (+ 1072us) log.cc:842] Serialized 1416 byte log entry
0620 23:01:27.580451 (+ 691us) raft_consensus.cc:1631] Marking committed up to 37
0620 23:01:27.581379 (+ 928us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:27.581416 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:27.581461 (+ 45us) raft_consensus.cc:1671] finished
0620 23:01:27.581465 (+ 4us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:27.581530 (+ 65us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:26.216941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:26.217147 (+ 206us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:26.217155 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:26.217157 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:26.217247 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:26.217260 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:26.217263 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:26.217311 (+ 48us) tablet.cc:699] Row locks acquired
0620 23:01:26.217312 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:27.580491 (+1363179us) op_driver.cc:444] REPLICATION: finished
0620 23:01:27.580966 (+ 475us) write_op.cc:301] APPLY: starting
0620 23:01:27.580986 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:27.581041 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:27.581045 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:27.581454 (+ 409us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:27.581489 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:27.581546 (+ 57us) write_op.cc:312] APPLY: finished
0620 23:01:27.581642 (+ 96us) log.cc:842] Serialized 22 byte log entry
0620 23:01:27.581726 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:27.581744 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:27.581750 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:27.581838 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:27.581845 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:26.767697 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:26.767823 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:26.767836 (+ 13us) write_op.cc:435] Acquired schema lock
0620 23:01:26.767839 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:26.767981 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:26.767997 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:26.768003 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:26.768067 (+ 64us) tablet.cc:699] Row locks acquired
0620 23:01:26.768069 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:27.580865 (+812796us) op_driver.cc:444] REPLICATION: finished
0620 23:01:27.581478 (+ 613us) write_op.cc:301] APPLY: starting
0620 23:01:27.581493 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:27.581530 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:27.581533 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:27.581624 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:27.581646 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:27.581868 (+ 222us) write_op.cc:312] APPLY: finished
0620 23:01:27.582005 (+ 137us) log.cc:842] Serialized 22 byte log entry
0620 23:01:27.582104 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:27.582118 (+ 14us) write_op.cc:454] Released schema lock
0620 23:01:27.582123 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:27.582220 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:27.582224 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:26.990310 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:26.990453 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:26.990459 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:26.990462 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:26.990566 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:26.990579 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:26.990582 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:26.990629 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:26.990630 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:01:27.579313 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:27.579422 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:27.579427 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:27.579429 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:27.579507 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:27.579519 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:27.579522 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:27.579581 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:01:27.579582 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":382,"threads_started":1,"wal-append.queue_time_us":731,"child_traces":[["op",{"apply.queue_time_us":419,"apply.run_cpu_time_us":912,"apply.run_wall_time_us":944,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":362,"prepare.run_cpu_time_us":565,"prepare.run_wall_time_us":554,"replication_time_us":2337205,"thread_start_us":533,"threads_started":2}],["op",{"apply.queue_time_us":558,"apply.run_cpu_time_us":613,"apply.run_wall_time_us":798,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":364,"prepare.run_cpu_time_us":459,"prepare.run_wall_time_us":459,"replication_time_us":1363816,"thread_start_us":708,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":412,"prepare.run_wall_time_us":413}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":480,"prepare.run_cpu_time_us":336,"prepare.run_wall_time_us":336,"thread_start_us":349,"threads_started":1}]]}
I20260620 23:01:27.583940 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:27.584214 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:28.249301 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:28.249576 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:28.250399 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:28.250928 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:28.250850 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:28.251559 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:28.251854 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:28.259476 13827 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=39, attempt_no=0}) took 1983 ms. Trace:
I20260620 23:01:28.259624 13827 rpcz_store.cc:276] 0620 23:01:26.276444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:26.276555 (+ 111us) service_pool.cc:227] Handling call
0620 23:01:28.259415 (+1982860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:26.630483 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:26.630665 (+ 182us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:26.630672 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:26.630674 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:26.630774 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:26.630786 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:26.630791 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:26.630853 (+ 62us) tablet.cc:699] Row locks acquired
0620 23:01:26.630855 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:26.630907 (+ 52us) write_op.cc:270] Start()
0620 23:01:26.814073 (+183166us) write_op.cc:276] Timestamp: P: 1781996486630925 usec, L: 0
0620 23:01:26.814079 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:26.814306 (+ 227us) log.cc:842] Serialized 354 byte log entry
0620 23:01:28.258513 (+1444207us) op_driver.cc:444] REPLICATION: finished
0620 23:01:28.258887 (+ 374us) write_op.cc:301] APPLY: starting
0620 23:01:28.258902 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:28.258932 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:28.258961 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:28.259022 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:28.259053 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:28.259057 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:28.259132 (+ 75us) log.cc:842] Serialized 22 byte log entry
0620 23:01:28.259265 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:28.259283 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:28.259292 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":322,"num_ops":1,"prepare.queue_time_us":353519,"prepare.run_cpu_time_us":861,"prepare.run_wall_time_us":183928,"replication_time_us":1444409,"thread_start_us":239,"threads_started":1}]]}
W20260620 23:01:28.601245 13220 tablet_service.cc:2970] error setting up scanner 14c77cd2ae3c4862867875bdda5b493b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1781996487600922 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1781996484519529 usec, L: 0 Physical time difference: 3.081s: new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
I20260620 23:01:28.601670 13220 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:49000 (request call id 75) took 1001 ms. Trace:
I20260620 23:01:28.601788 13220 rpcz_store.cc:276] 0620 23:01:27.599728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:27.599805 (+ 77us) service_pool.cc:227] Handling call
0620 23:01:27.600072 (+ 267us) tablet_service.cc:2780] Created scanner 14c77cd2ae3c4862867875bdda5b493b for tablet cb4f22b70c6e41e49ebfc29f973106da, query id is 6af82449cbc046db87b2b0e0eb491c3e
0620 23:01:27.600878 (+ 806us) tablet_service.cc:2920] Creating iterator
0620 23:01:27.600909 (+ 31us) tablet_service.cc:3297] Waiting safe time to advance
0620 23:01:28.601177 (+1000268us) tablet_service.cc:2945] Iterator created
0620 23:01:28.601193 (+ 16us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1781996487600922 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1781996484519529 usec, L: 0 Physical time difference: 3.081s
0620 23:01:28.601641 (+ 448us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260620 23:01:28.755038 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:28.755232 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:28.756019 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:28.756721 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:28.756768 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:28.757377 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:28.757649 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:28.802031 13833 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=40, attempt_no=0}) took 1422 ms. Trace:
I20260620 23:01:28.802196 13833 rpcz_store.cc:276] 0620 23:01:27.379195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:27.379275 (+ 80us) service_pool.cc:227] Handling call
0620 23:01:28.801914 (+1422639us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:27.380052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:27.380170 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:27.380173 (+ 3us) write_op.cc:435] Acquired schema lock
0620 23:01:27.380174 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:27.380252 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:27.380261 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:27.380264 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:27.380335 (+ 71us) tablet.cc:699] Row locks acquired
0620 23:01:27.380337 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:27.380368 (+ 31us) write_op.cc:270] Start()
0620 23:01:27.735564 (+355196us) write_op.cc:276] Timestamp: P: 1781996487380289 usec, L: 0
0620 23:01:27.735570 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:27.735834 (+ 264us) log.cc:842] Serialized 354 byte log entry
0620 23:01:28.800790 (+1064956us) op_driver.cc:444] REPLICATION: finished
0620 23:01:28.801306 (+ 516us) write_op.cc:301] APPLY: starting
0620 23:01:28.801327 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:28.801374 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:28.801408 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:28.801534 (+ 126us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:28.801574 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:28.801580 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:28.801703 (+ 123us) log.cc:842] Serialized 22 byte log entry
0620 23:01:28.801794 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:28.801814 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:28.801821 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":423,"num_ops":1,"prepare.queue_time_us":359,"prepare.run_cpu_time_us":801,"prepare.run_wall_time_us":355868,"replication_time_us":1065189,"thread_start_us":589,"threads_started":2}]]}
I20260620 23:01:29.101811 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:29.101852 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 213) took 1517 ms. Trace:
I20260620 23:01:29.102064 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:29.102885 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:29.102068 13262 rpcz_store.cc:276] 0620 23:01:27.584843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:27.584899 (+ 56us) service_pool.cc:227] Handling call
0620 23:01:27.585082 (+ 183us) raft_consensus.cc:1457] Updating replica for 3 ops
0620 23:01:27.585186 (+ 104us) raft_consensus.cc:1520] Early marking committed up to index 39
0620 23:01:27.585286 (+ 100us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0620 23:01:27.585397 (+ 111us) write_op.cc:270] Start()
0620 23:01:28.307589 (+722192us) write_op.cc:276] Timestamp: P: 1781996485424913 usec, L: 0
0620 23:01:28.308129 (+ 540us) write_op.cc:270] Start()
0620 23:01:28.947289 (+639160us) write_op.cc:276] Timestamp: P: 1781996486136849 usec, L: 0
0620 23:01:28.947797 (+ 508us) write_op.cc:270] Start()
0620 23:01:29.099963 (+152166us) write_op.cc:276] Timestamp: P: 1781996486630925 usec, L: 0
0620 23:01:29.100448 (+ 485us) log.cc:842] Serialized 1062 byte log entry
0620 23:01:29.100806 (+ 358us) raft_consensus.cc:1631] Marking committed up to 41
0620 23:01:29.101644 (+ 838us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:29.101683 (+ 39us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:29.101692 (+ 9us) raft_consensus.cc:1671] finished
0620 23:01:29.101695 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:29.101799 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:28.308139 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:28.308262 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:28.308267 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:28.308269 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:28.308379 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:28.308399 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:28.308403 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:28.308466 (+ 63us) tablet.cc:699] Row locks acquired
0620 23:01:28.308468 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:29.100837 (+792369us) op_driver.cc:444] REPLICATION: finished
0620 23:01:29.101270 (+ 433us) write_op.cc:301] APPLY: starting
0620 23:01:29.101288 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:29.101331 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:29.101337 (+ 6us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:29.101461 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:29.101512 (+ 51us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:29.101666 (+ 154us) write_op.cc:312] APPLY: finished
0620 23:01:29.101754 (+ 88us) log.cc:842] Serialized 22 byte log entry
0620 23:01:29.102028 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:29.102050 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:29.102063 (+ 13us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:29.102172 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:29.102176 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:28.947754 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:28.947859 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:28.947864 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:28.947866 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:28.947942 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:28.947954 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:28.947957 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:28.948002 (+ 45us) tablet.cc:699] Row locks acquired
0620 23:01:28.948003 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:29.101192 (+153189us) op_driver.cc:444] REPLICATION: finished
0620 23:01:29.102513 (+ 1321us) write_op.cc:301] APPLY: starting
Related trace 'op':
0620 23:01:29.100198 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:29.100307 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:29.100313 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:29.100315 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:29.100393 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:29.100405 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:29.100409 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:29.100457 (+ 48us) tablet.cc:699] Row locks acquired
0620 23:01:29.100458 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":266,"threads_started":1,"wal-append.queue_time_us":377,"child_traces":[["op",{"apply.queue_time_us":374,"apply.run_cpu_time_us":686,"apply.run_wall_time_us":968,"num_ops":1,"partition_lock_wait_us":9,"prepare.queue_time_us":394,"prepare.run_cpu_time_us":490,"prepare.run_wall_time_us":490,"replication_time_us":1515429,"thread_start_us":505,"threads_started":2}],["op",{"apply.queue_time_us":524,"mutex_wait_us":44,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":334,"prepare.run_cpu_time_us":332,"prepare.run_wall_time_us":333,"replication_time_us":793049,"spinlock_wait_cycles":11136,"thread_start_us":537,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":83,"prepare.run_cpu_time_us":329,"prepare.run_wall_time_us":330}]]}
I20260620 23:01:29.103659 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:29.103785 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:29.104420 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:29.166754 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:29.167008 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:29.167853 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:29.168402 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:29.168562 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:29.169147 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:29.762792 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:30.482158 13220 tablet_service.cc:2970] error setting up scanner f05b1310513840dd8d3eb150d8953ff3 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.377s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:30.786463 13740 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:30.786737 13740 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:30.787582 13740 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:30.788113 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:30.788177 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:30.788870 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:30.822504 13850 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=43, attempt_no=0}) took 1608 ms. Trace:
I20260620 23:01:30.822640 13850 rpcz_store.cc:276] 0620 23:01:29.214142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:29.214220 (+ 78us) service_pool.cc:227] Handling call
0620 23:01:30.822447 (+1608227us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:29.214824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:29.214917 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:29.214921 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:29.214922 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:29.215001 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:29.215010 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:29.215014 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:29.215053 (+ 39us) tablet.cc:699] Row locks acquired
0620 23:01:29.215055 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:29.215086 (+ 31us) write_op.cc:270] Start()
0620 23:01:30.082255 (+867169us) write_op.cc:276] Timestamp: P: 1781996489215099 usec, L: 0
0620 23:01:30.082261 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:30.082480 (+ 219us) log.cc:842] Serialized 354 byte log entry
0620 23:01:30.821346 (+738866us) op_driver.cc:444] REPLICATION: finished
0620 23:01:30.821836 (+ 490us) write_op.cc:301] APPLY: starting
0620 23:01:30.821855 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:30.821929 (+ 74us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:30.822034 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:30.822135 (+ 101us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:30.822165 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:30.822170 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:30.822264 (+ 94us) log.cc:842] Serialized 22 byte log entry
0620 23:01:30.822333 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:30.822349 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:30.822355 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":422,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":867749,"replication_time_us":739046,"thread_start_us":287,"threads_started":1}]]}
I20260620 23:01:30.858366 13740 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:30.868177 13850 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=44, attempt_no=0}) took 1085 ms. Trace:
I20260620 23:01:30.868337 13850 rpcz_store.cc:276] 0620 23:01:29.783058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:29.783136 (+ 78us) service_pool.cc:227] Handling call
0620 23:01:30.868122 (+1084986us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:30.082617 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:30.082738 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:30.082745 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:30.082747 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:30.082897 (+ 150us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:30.082912 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:30.082919 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:30.082977 (+ 58us) tablet.cc:699] Row locks acquired
0620 23:01:30.082978 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:30.083016 (+ 38us) write_op.cc:270] Start()
0620 23:01:30.153170 (+ 70154us) write_op.cc:276] Timestamp: P: 1781996490083035 usec, L: 0
0620 23:01:30.153175 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:30.153473 (+ 298us) log.cc:842] Serialized 354 byte log entry
0620 23:01:30.867446 (+713973us) op_driver.cc:444] REPLICATION: finished
0620 23:01:30.867550 (+ 104us) write_op.cc:301] APPLY: starting
0620 23:01:30.867575 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:30.867614 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:30.867643 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:30.867733 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:30.867774 (+ 41us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:30.867780 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:30.867869 (+ 89us) log.cc:842] Serialized 22 byte log entry
0620 23:01:30.868005 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:30.868020 (+ 15us) write_op.cc:454] Released schema lock
0620 23:01:30.868026 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"num_ops":1,"prepare.queue_time_us":298961,"prepare.run_cpu_time_us":869,"prepare.run_wall_time_us":70938,"replication_time_us":714245}]]}
I20260620 23:01:31.256242 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:31.256718 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 214) took 2152 ms. Trace:
I20260620 23:01:31.256862 13262 rpcz_store.cc:276] 0620 23:01:29.104218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:29.104266 (+ 48us) service_pool.cc:227] Handling call
0620 23:01:29.104439 (+ 173us) raft_consensus.cc:1457] Updating replica for 3 ops
0620 23:01:29.104644 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 42
0620 23:01:29.104757 (+ 113us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0620 23:01:29.104867 (+ 110us) write_op.cc:270] Start()
0620 23:01:29.888043 (+783176us) write_op.cc:276] Timestamp: P: 1781996487380289 usec, L: 0
0620 23:01:29.888687 (+ 644us) write_op.cc:270] Start()
0620 23:01:30.641858 (+753171us) write_op.cc:276] Timestamp: P: 1781996488262806 usec, L: 0
0620 23:01:30.642484 (+ 626us) write_op.cc:270] Start()
0620 23:01:31.254624 (+612140us) write_op.cc:276] Timestamp: P: 1781996488884733 usec, L: 0
0620 23:01:31.255402 (+ 778us) log.cc:842] Serialized 1062 byte log entry
0620 23:01:31.255773 (+ 371us) raft_consensus.cc:1631] Marking committed up to 43
0620 23:01:31.256116 (+ 343us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:31.256153 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:31.256593 (+ 440us) raft_consensus.cc:1671] finished
0620 23:01:31.256595 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:31.256670 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:29.888620 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:29.888771 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:29.888778 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:29.888780 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:29.888880 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:29.888897 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:29.888904 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:29.889004 (+ 100us) tablet.cc:699] Row locks acquired
0620 23:01:29.889007 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:31.255811 (+1366804us) op_driver.cc:444] REPLICATION: finished
0620 23:01:31.256348 (+ 537us) write_op.cc:301] APPLY: starting
0620 23:01:31.256366 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:31.256409 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:31.256412 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:31.256484 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:31.256520 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:31.256552 (+ 32us) write_op.cc:312] APPLY: finished
0620 23:01:31.256678 (+ 126us) log.cc:842] Serialized 22 byte log entry
0620 23:01:31.256746 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:31.256761 (+ 15us) write_op.cc:454] Released schema lock
0620 23:01:31.256768 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:31.256850 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:31.256854 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:30.642430 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:30.642577 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:30.642586 (+ 9us) write_op.cc:435] Acquired schema lock
0620 23:01:30.642590 (+ 4us) tablet.cc:660] Decoding operations
0620 23:01:30.642716 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:30.642733 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:30.642739 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:30.642796 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:01:30.642798 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:01:31.255204 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:31.255346 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:31.255354 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:31.255357 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:31.255460 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:31.255476 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:31.255480 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:31.255540 (+ 60us) tablet.cc:699] Row locks acquired
0620 23:01:31.255544 (+ 4us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":273,"threads_started":1,"wal-append.queue_time_us":363,"child_traces":[["op",{"apply.queue_time_us":359,"apply.run_cpu_time_us":577,"apply.run_wall_time_us":688,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":436,"prepare.run_cpu_time_us":489,"prepare.run_wall_time_us":489,"replication_time_us":2150929,"thread_start_us":548,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":364,"prepare.run_cpu_time_us":476,"prepare.run_wall_time_us":475,"thread_start_us":232,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":397,"prepare.run_cpu_time_us":421,"prepare.run_wall_time_us":422,"thread_start_us":279,"threads_started":1}]]}
W20260620 23:01:31.932824 13220 tablet_service.cc:2970] error setting up scanner ceba2331bcbc47c0bee6edd3871503dd with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.674s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "6af82449cbc046db87b2b0e0eb491c3e"
I20260620 23:01:32.348363 13856 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:32.348621 13856 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:32.349572 13856 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:32.350152 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:32.350347 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:32.350940 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:32.367054 13863 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=45, attempt_no=0}) took 1542 ms. Trace:
I20260620 23:01:32.367184 13863 rpcz_store.cc:276] 0620 23:01:30.824519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:30.824583 (+ 64us) service_pool.cc:227] Handling call
0620 23:01:32.366977 (+1542394us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:30.825330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:30.825450 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:30.825454 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:30.825455 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:30.825571 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:30.825579 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:30.825583 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:30.825667 (+ 84us) tablet.cc:699] Row locks acquired
0620 23:01:30.825669 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:30.825706 (+ 37us) write_op.cc:270] Start()
0620 23:01:31.556910 (+731204us) write_op.cc:276] Timestamp: P: 1781996490825690 usec, L: 0
0620 23:01:31.556918 (+ 8us) op_driver.cc:331] REPLICATION: starting
0620 23:01:31.557125 (+ 207us) log.cc:842] Serialized 354 byte log entry
0620 23:01:32.366016 (+808891us) op_driver.cc:444] REPLICATION: finished
0620 23:01:32.366506 (+ 490us) write_op.cc:301] APPLY: starting
0620 23:01:32.366526 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:32.366560 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:32.366589 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:32.366661 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:32.366688 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:32.366692 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:32.366797 (+ 105us) log.cc:842] Serialized 22 byte log entry
0620 23:01:32.366865 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:32.366881 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:32.366887 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":430,"num_ops":1,"prepare.queue_time_us":342,"prepare.run_cpu_time_us":813,"prepare.run_wall_time_us":731887,"replication_time_us":809008,"thread_start_us":523,"threads_started":2}]]}
I20260620 23:01:32.958320 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:32.958353 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 215) took 1699 ms. Trace:
I20260620 23:01:32.958563 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:32.958562 13262 rpcz_store.cc:276] 0620 23:01:31.258566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:31.258650 (+ 84us) service_pool.cc:227] Handling call
0620 23:01:31.258838 (+ 188us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:31.258985 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 45
0620 23:01:31.259400 (+ 415us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:31.259528 (+ 128us) write_op.cc:270] Start()
0620 23:01:32.181694 (+922166us) write_op.cc:276] Timestamp: P: 1781996489215099 usec, L: 0
0620 23:01:32.182244 (+ 550us) write_op.cc:270] Start()
0620 23:01:32.956408 (+774164us) write_op.cc:276] Timestamp: P: 1781996490083035 usec, L: 0
0620 23:01:32.957079 (+ 671us) log.cc:842] Serialized 708 byte log entry
0620 23:01:32.957415 (+ 336us) raft_consensus.cc:1631] Marking committed up to 47
0620 23:01:32.958199 (+ 784us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:32.958234 (+ 35us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:32.958244 (+ 10us) raft_consensus.cc:1671] finished
0620 23:01:32.958247 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:32.958310 (+ 63us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:32.182202 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:32.182314 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:32.182318 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:32.182320 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:32.182398 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:32.182411 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:32.182414 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:32.182460 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:01:32.182462 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:32.957456 (+774994us) op_driver.cc:444] REPLICATION: finished
0620 23:01:32.957818 (+ 362us) write_op.cc:301] APPLY: starting
0620 23:01:32.957830 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:32.957861 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:32.957863 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:32.958004 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:32.958057 (+ 53us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:32.958117 (+ 60us) write_op.cc:312] APPLY: finished
0620 23:01:32.958228 (+ 111us) log.cc:842] Serialized 22 byte log entry
0620 23:01:32.958300 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:32.958316 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:32.958322 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:32.958398 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:32.958402 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:01:32.956867 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:32.957005 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:32.957013 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:32.957017 (+ 4us) tablet.cc:660] Decoding operations
0620 23:01:32.957109 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:32.957121 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:32.957125 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:32.957169 (+ 44us) tablet.cc:699] Row locks acquired
0620 23:01:32.957171 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:32.957737 (+ 566us) op_driver.cc:444] REPLICATION: finished
0620 23:01:32.958307 (+ 570us) write_op.cc:301] APPLY: starting
0620 23:01:32.958317 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:32.958349 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:32.958351 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:32.958448 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:32.958482 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:32.958560 (+ 78us) write_op.cc:312] APPLY: finished
0620 23:01:32.958660 (+ 100us) log.cc:842] Serialized 22 byte log entry
0620 23:01:32.958700 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:32.958715 (+ 15us) write_op.cc:454] Released schema lock
0620 23:01:32.958720 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0620 23:01:32.958779 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:32.958783 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":245,"threads_started":1,"wal-append.queue_time_us":353,"child_traces":[["op",{"apply.queue_time_us":295,"apply.run_cpu_time_us":647,"apply.run_wall_time_us":646,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":371,"prepare.run_cpu_time_us":363,"prepare.run_wall_time_us":363,"replication_time_us":1697911,"thread_start_us":436,"threads_started":2}],["op",{"apply.queue_time_us":510,"apply.run_cpu_time_us":538,"apply.run_wall_time_us":539,"mutex_wait_us":77,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":329,"prepare.run_cpu_time_us":417,"prepare.run_wall_time_us":417,"replication_time_us":775481,"thread_start_us":525,"threads_started":2}]]}
I20260620 23:01:32.959555 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:32.960042 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 47 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:32.960068 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 47 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:32.960704 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:32.960989 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:33.281766 13856 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:33.295356 13872 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34484 (ReqId={client: e0452562fbaa4d0b9bcb532ed058ed95, seq_no=46, attempt_no=0}) took 2425 ms. Trace:
I20260620 23:01:33.295513 13872 rpcz_store.cc:276] 0620 23:01:30.870046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:30.870178 (+ 132us) service_pool.cc:227] Handling call
0620 23:01:33.295288 (+2425110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:31.557282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:31.557414 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:31.557422 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:31.557424 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:31.557519 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:31.557532 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:31.557540 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:31.557627 (+ 87us) tablet.cc:699] Row locks acquired
0620 23:01:31.557630 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:31.557680 (+ 50us) write_op.cc:270] Start()
0620 23:01:31.757838 (+200158us) write_op.cc:276] Timestamp: P: 1781996491557731 usec, L: 0
0620 23:01:31.757844 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:31.758210 (+ 366us) log.cc:842] Serialized 354 byte log entry
0620 23:01:33.294178 (+1535968us) op_driver.cc:444] REPLICATION: finished
0620 23:01:33.294611 (+ 433us) write_op.cc:301] APPLY: starting
0620 23:01:33.294629 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:33.294705 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:33.294743 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:33.294830 (+ 87us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:33.294860 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:33.294864 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:33.294991 (+ 127us) log.cc:842] Serialized 22 byte log entry
0620 23:01:33.295156 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:33.295178 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:33.295186 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":371,"num_ops":1,"prepare.queue_time_us":686598,"prepare.run_cpu_time_us":959,"prepare.run_wall_time_us":201030,"replication_time_us":1536303,"thread_start_us":267,"threads_started":1}]]}
W20260620 23:01:33.301033 13220 tablet_service.cc:2970] error setting up scanner 321e379e94d54f02a6a97a9cad726245 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.343s (maximum allowed 0.100s): new_scan_request { tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7299057566185213952 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\332\271\334\321\006\"3\n\005slave\022*\n a2a5ccb43a764745b7815fdefad32428\020\001\030\001 \001(\001" signature: "<redacted>""\263\013Pr\345\346\301x\034\231\261\0343h<8\240\300\226\372,\025jX\204\024El\304\256\206\324\2405\2475\321\264\324\374\222\204\\\206\006\033\223\266\366\203\313\333V\013\276\323\367\260\312\021\354\006mS" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "75667c2a22344d7997e9cf54f6ac02be"
I20260620 23:01:34.244463 13717 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:34.244463 13262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.130:45955 (request call id 216) took 1283 ms. Trace:
I20260620 23:01:34.244702 13717 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:34.244681 13262 rpcz_store.cc:276] 0620 23:01:32.961310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:32.961379 (+ 69us) service_pool.cc:227] Handling call
0620 23:01:32.961593 (+ 214us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:32.961715 (+ 122us) raft_consensus.cc:1520] Early marking committed up to index 0
0620 23:01:32.961718 (+ 3us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:32.961816 (+ 98us) write_op.cc:270] Start()
0620 23:01:33.688976 (+727160us) write_op.cc:276] Timestamp: P: 1781996490825690 usec, L: 0
0620 23:01:33.690313 (+ 1337us) write_op.cc:270] Start()
0620 23:01:34.242457 (+552144us) write_op.cc:276] Timestamp: P: 1781996491557731 usec, L: 0
0620 23:01:34.243166 (+ 709us) log.cc:842] Serialized 708 byte log entry
0620 23:01:34.243617 (+ 451us) raft_consensus.cc:1631] Marking committed up to 48
0620 23:01:34.244298 (+ 681us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:34.244336 (+ 38us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:34.244347 (+ 11us) raft_consensus.cc:1671] finished
0620 23:01:34.244350 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:34.244397 (+ 47us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:33.690210 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:33.690335 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:33.690344 (+ 9us) write_op.cc:435] Acquired schema lock
0620 23:01:33.690347 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:33.690469 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:33.690488 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:33.690493 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:33.690554 (+ 61us) tablet.cc:699] Row locks acquired
0620 23:01:33.690556 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:34.243656 (+553100us) op_driver.cc:444] REPLICATION: finished
0620 23:01:34.244396 (+ 740us) write_op.cc:301] APPLY: starting
0620 23:01:34.244414 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:34.244466 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:34.244468 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:34.244550 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:34.244585 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:34.244626 (+ 41us) write_op.cc:312] APPLY: finished
0620 23:01:34.244719 (+ 93us) log.cc:842] Serialized 22 byte log entry
Related trace 'op':
0620 23:01:34.243030 (+ 0us) write_op.cc:183] PREPARE: starting on tablet cb4f22b70c6e41e49ebfc29f973106da
0620 23:01:34.243158 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:34.243165 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:34.243168 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:34.243275 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:34.243310 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:34.243315 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:34.243383 (+ 68us) tablet.cc:699] Row locks acquired
0620 23:01:34.243385 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":356,"threads_started":1,"wal-append.queue_time_us":467,"child_traces":[["op",{"apply.queue_time_us":662,"apply.run_cpu_time_us":570,"apply.run_wall_time_us":743,"num_ops":1,"partition_lock_wait_us":8,"prepare.queue_time_us":1091,"prepare.run_cpu_time_us":468,"prepare.run_wall_time_us":467,"replication_time_us":1281823,"thread_start_us":1536,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":372,"prepare.run_cpu_time_us":484,"prepare.run_wall_time_us":481,"thread_start_us":241,"threads_started":1}]]}
I20260620 23:01:34.245579 13717 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:34.246436 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:34.246881 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:34.247768 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:34.248163 13717 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:34.498993 13615 scanner-internal.cc:458] Time spent opening tablet: real 18.228s user 0.000s sys 0.007s
I20260620 23:01:35.146521 13879 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:35.146653 13880 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:35.146903 13879 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.146859 13880 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.148033 13880 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:35.148172 13879 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:35.148532 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:35.148787 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:35.149415 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:35.150079 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:35.150637 13261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:35.150837 13261 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb45f804a3ff4b12a653ee059501b6f5 in term 3.
I20260620 23:01:35.150995 13879 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:35.151391 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539, eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45
I20260620 23:01:35.151708 13880 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 23:01:35.151830 13880 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:35.151933 13880 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 3 FOLLOWER]: Advancing to term 4
I20260620 23:01:35.154481 13880 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.154836 13522 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Rejecting Update request from peer 6a46fb3b701f43cb96ea5c4958cb0a45 for earlier term 3. Current term is 4. Ops: []
I20260620 23:01:35.155274 13880 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 election: Requested vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:35.155673 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539"
I20260620 23:01:35.155793 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
I20260620 23:01:35.156035 13878 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: INVALID_TERM, Last received: 3.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260620 23:01:35.156579 13458 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:35.156903 13880 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260620 23:01:35.158151 13522 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Rejecting Update request from peer 6a46fb3b701f43cb96ea5c4958cb0a45 for earlier term 3. Current term is 4. Ops: []
I20260620 23:01:35.158669 13878 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: INVALID_TERM, Last received: 3.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260620 23:01:35.160427 13522 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Rejecting Update request from peer 6a46fb3b701f43cb96ea5c4958cb0a45 for earlier term 3. Current term is 4. Ops: []
I20260620 23:01:35.160979 13870 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: INVALID_TERM, Last received: 3.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260620 23:01:35.162804 13522 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Rejecting Update request from peer 6a46fb3b701f43cb96ea5c4958cb0a45 for earlier term 3. Current term is 4. Ops: []
I20260620 23:01:35.163511 13880 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:35.163358 13870 consensus_queue.cc:1046] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: INVALID_TERM, Last received: 3.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260620 23:01:35.163661 13880 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.164039 13880 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:35.164497 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:35.164549 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:35.165043 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:35.165282 13880 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260620 23:01:35.165437 13522 raft_consensus.cc:1226] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Rejecting Update request from peer 6a46fb3b701f43cb96ea5c4958cb0a45 for earlier term 3. Current term is 4. Ops: []
I20260620 23:01:35.165683 13877 raft_consensus.cc:3032] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 LEADER]: Stepping down as leader of term 3
I20260620 23:01:35.165794 13877 raft_consensus.cc:729] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6a46fb3b701f43cb96ea5c4958cb0a45, State: Running, Role: LEADER
I20260620 23:01:35.166190 13877 consensus_queue.cc:260] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 3.49, Last appended by leader: 49, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.166597 13877 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 3 FOLLOWER]: Advancing to term 4
I20260620 23:01:35.171835 13878 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:35.171957 13878 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.172379 13879 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:35.172526 13879 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.172870 13878 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:35.173280 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:35.173537 13262 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 3.
I20260620 23:01:35.173503 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:35.173543 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:35.173724 13392 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dce4229f6f247c7b2b68b3a75c39539 in term 4.
I20260620 23:01:35.173816 13521 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 in current term 4: Already voted for candidate eb45f804a3ff4b12a653ee059501b6f5 in this term.
I20260620 23:01:35.173920 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539; no voters:
I20260620 23:01:35.174140 13879 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:35.174175 13196 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539; no voters:
I20260620 23:01:35.173535 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:35.174433 13878 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260620 23:01:35.174458 13879 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 23:01:35.174475 13522 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 4.
I20260620 23:01:35.174537 13878 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:35.174562 13879 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:35.174615 13878 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 4 FOLLOWER]: Advancing to term 5
I20260620 23:01:35.174664 13879 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 3 FOLLOWER]: Advancing to term 4
I20260620 23:01:35.176393 13878 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.176394 13879 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.176999 13879 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 election: Requested vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:35.177280 13878 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 5 election: Requested vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
I20260620 23:01:35.177343 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5"
I20260620 23:01:35.177305 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539"
I20260620 23:01:35.177407 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45"
I20260620 23:01:35.177585 13522 raft_consensus.cc:2379] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 in current term 4: Already voted for candidate eb45f804a3ff4b12a653ee059501b6f5 in this term.
I20260620 23:01:35.177659 13392 raft_consensus.cc:2354] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 4. Current term is 5.
I20260620 23:01:35.177626 13262 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 4 FOLLOWER]: Advancing to term 5
I20260620 23:01:35.178165 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5"
I20260620 23:01:35.178221 13196 leader_election.cc:398] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 election: Vote denied by peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257) with higher term. Message: Invalid argument: T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 4. Current term is 5.
I20260620 23:01:35.178308 13196 leader_election.cc:401] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260620 23:01:35.178339 13522 raft_consensus.cc:3037] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 4 FOLLOWER]: Advancing to term 5
I20260620 23:01:35.179942 13262 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 5.
I20260620 23:01:35.180166 13879 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257) with higher term. Message: Invalid argument: T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dce4229f6f247c7b2b68b3a75c39539 for earlier term 4. Current term is 5.
I20260620 23:01:35.180241 13522 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a46fb3b701f43cb96ea5c4958cb0a45 in term 5.
I20260620 23:01:35.180315 13328 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539; no voters:
I20260620 23:01:35.180605 13878 raft_consensus.cc:2781] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 5 FOLLOWER]: Leader election won for term 5
I20260620 23:01:35.180752 13878 raft_consensus.cc:686] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [term 5 LEADER]: Becoming Leader. State: Replica: 6a46fb3b701f43cb96ea5c4958cb0a45, State: Running, Role: LEADER
I20260620 23:01:35.181018 13878 consensus_queue.cc:237] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 3.49, Last appended by leader: 49, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:35.183406 13522 raft_consensus.cc:1261] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Refusing update from remote peer 6a46fb3b701f43cb96ea5c4958cb0a45: Log matching property violated. Preceding OpId in replica: term: 3 index: 49. Preceding OpId from leader: term: 5 index: 50. (index mismatch)
I20260620 23:01:35.183545 13262 raft_consensus.cc:1261] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Refusing update from remote peer 6a46fb3b701f43cb96ea5c4958cb0a45: Log matching property violated. Preceding OpId in replica: term: 3 index: 49. Preceding OpId from leader: term: 5 index: 50. (index mismatch)
I20260620 23:01:35.184145 13878 consensus_queue.cc:1035] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20260620 23:01:35.183933 13119 catalog_manager.cc:5495] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:35.184454 13870 consensus_queue.cc:1035] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
W20260620 23:01:35.220022 13878 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 5 is higher.
W20260620 23:01:35.240289 13878 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 5 is higher.
W20260620 23:01:35.253636 13878 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 5 is higher.
W20260620 23:01:36.124708 13616 scanner-internal.cc:458] Time spent opening tablet: real 19.854s user 0.008s sys 0.000s
I20260620 23:01:36.236032 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.237188 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.240026 13914 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:36.246136 13456 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.134.130:38257 timed out after 0.006s (SENT) [suppressed 1 similar messages]
I20260620 23:01:36.248535 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.251961 13915 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.252149 13915 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.252754 13915 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:36.253230 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.253306 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:36.253580 13262 raft_consensus.cc:2454] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb45f804a3ff4b12a653ee059501b6f5 in term 5.
W20260620 23:01:36.259050 13195 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.134.131:40719 timed out after 0.006s (SENT) [suppressed 1 similar messages]
I20260620 23:01:36.267831 13372 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260620 23:01:36.275107 13242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20260620 23:01:36.282097 13195 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719): Timed out: RequestConsensusVote RPC to 127.10.134.131:40719 timed out after 0.006s (SENT)
I20260620 23:01:36.282338 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:36.282662 13915 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.293874 13915 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.294075 13915 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.294770 13915 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719), 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257)
I20260620 23:01:36.295281 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
W20260620 23:01:36.306792 13456 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.10.134.130:38257, user_credentials={real_user=slave}} blocked reactor thread for 60644.1us
W20260620 23:01:36.310070 13195 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer eb45f804a3ff4b12a653ee059501b6f5 (127.10.134.131:40719): Timed out: RequestConsensusVote RPC to 127.10.134.131:40719 timed out after 0.007s (SENT)
I20260620 23:01:36.310225 13195 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dce4229f6f247c7b2b68b3a75c39539; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, eb45f804a3ff4b12a653ee059501b6f5
I20260620 23:01:36.310451 13915 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.310868 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.311630 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6dce4229f6f247c7b2b68b3a75c39539" is_pre_election: true
I20260620 23:01:36.313290 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:36.313728 13914 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.322525 13522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:36.322647 13521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:36.322791 13520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:36.323817 13520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "6dce4229f6f247c7b2b68b3a75c39539" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "eb45f804a3ff4b12a653ee059501b6f5" is_pre_election: true
I20260620 23:01:36.335587 13502 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
284d9dcd534240888dba87e55c1b2ebd | 127.10.134.190:34675 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38069 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
6a46fb3b701f43cb96ea5c4958cb0a45 | 127.10.134.130:38257 | HEALTHY | <none> | 1 | 0
6dce4229f6f247c7b2b68b3a75c39539 | 127.10.134.129:36713 | HEALTHY | <none> | 0 | 0
eb45f804a3ff4b12a653ee059501b6f5 | 127.10.134.131:40719 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:36713
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:38257
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:40719
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:36713
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:38257
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:40719
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:38069 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260620 23:01:36.375636 10778 log_verifier.cc:126] Checking tablet cb4f22b70c6e41e49ebfc29f973106da
I20260620 23:01:36.436851 10778 log_verifier.cc:177] Verified matching terms for 50 ops in tablet cb4f22b70c6e41e49ebfc29f973106da
I20260620 23:01:36.469655 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 13180
I20260620 23:01:36.479036 13302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:01:36.652001 13915 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.652227 13915 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P 6dce4229f6f247c7b2b68b3a75c39539 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.652730 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 13180
W20260620 23:01:36.682732 13328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.134.129:36713 (error 108) [suppressed 1 similar messages]
I20260620 23:01:36.683362 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 13310
W20260620 23:01:36.685307 13328 consensus_peers.cc:487] T cb4f22b70c6e41e49ebfc29f973106da P 6a46fb3b701f43cb96ea5c4958cb0a45 -> Peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Couldn't send request to peer 6dce4229f6f247c7b2b68b3a75c39539. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:36713: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:01:36.690716 13432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:01:36.696571 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.696732 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.697418 13914 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:36.699399 13458 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Network error: Client connection negotiation failed: client connection to 127.10.134.129:36713: connect: Connection refused (error 111)
W20260620 23:01:36.710307 13456 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257): Timed out: RequestConsensusVote RPC to 127.10.134.130:38257 timed out after 0.006s (SENT)
I20260620 23:01:36.710467 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:36.710717 13914 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.718454 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.718622 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.719231 13914 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:36.720517 13458 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Network error: Client connection negotiation failed: client connection to 127.10.134.129:36713: connect: Connection refused (error 111)
W20260620 23:01:36.742070 13456 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257): Timed out: RequestConsensusVote RPC to 127.10.134.130:38257 timed out after 0.011s (SENT)
I20260620 23:01:36.742206 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:36.742475 13914 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.756475 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.756644 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.757256 13914 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:36.758637 13458 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Network error: Client connection negotiation failed: client connection to 127.10.134.129:36713: connect: Connection refused (error 111)
W20260620 23:01:36.786443 13456 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257): Timed out: RequestConsensusVote RPC to 127.10.134.130:38257 timed out after 0.014s (SENT)
I20260620 23:01:36.786605 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:36.786897 13914 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.793754 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.794023 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.794708 13914 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:36.796157 13458 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Network error: Client connection negotiation failed: client connection to 127.10.134.129:36713: connect: Connection refused (error 111)
I20260620 23:01:36.837407 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.838605 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.838860 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.839138 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.840042 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.840718 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:36.839797 13390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4f22b70c6e41e49ebfc29f973106da" candidate_uuid: "eb45f804a3ff4b12a653ee059501b6f5" candidate_term: 6 candidate_status { last_received { term: 5 index: 50 } } ignore_live_leader: false dest_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" is_pre_election: true
I20260620 23:01:36.841492 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 13310
I20260620 23:01:36.843149 13914 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.850871 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:36.851050 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:36.851655 13914 leader_election.cc:288] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257), 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713)
W20260620 23:01:36.853101 13458 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6dce4229f6f247c7b2b68b3a75c39539 (127.10.134.129:36713): Network error: Client connection negotiation failed: client connection to 127.10.134.129:36713: connect: Connection refused (error 111)
W20260620 23:01:36.864006 13456 connection.cc:535] client connection to 127.10.134.130:38257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 23:01:36.864127 13456 leader_election.cc:334] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6a46fb3b701f43cb96ea5c4958cb0a45 (127.10.134.130:38257): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:01:36.864287 13456 leader_election.cc:302] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb45f804a3ff4b12a653ee059501b6f5; no voters: 6a46fb3b701f43cb96ea5c4958cb0a45, 6dce4229f6f247c7b2b68b3a75c39539
I20260620 23:01:36.864358 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 13440
I20260620 23:01:36.864545 13914 raft_consensus.cc:2726] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260620 23:01:36.870287 13562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:01:37.003240 13914 raft_consensus.cc:482] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6a46fb3b701f43cb96ea5c4958cb0a45)
I20260620 23:01:37.003456 13914 raft_consensus.cc:504] T cb4f22b70c6e41e49ebfc29f973106da P eb45f804a3ff4b12a653ee059501b6f5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb45f804a3ff4b12a653ee059501b6f5" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 40719 } } peers { permanent_uuid: "6a46fb3b701f43cb96ea5c4958cb0a45" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38257 } } peers { permanent_uuid: "6dce4229f6f247c7b2b68b3a75c39539" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 36713 } }
I20260620 23:01:37.003878 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 13440
I20260620 23:01:37.030938 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 13090
I20260620 23:01:37.032013 13151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:01:37.139850 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 13090
2026-06-20T23:01:37Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (36892 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20260620 23:01:37.183604 10778 test_util.cc:268] Using random seed: 1681171294
I20260620 23:01:37.185271 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:01:37.185328 10778 ts_itest-base.cc:113] --------------
I20260620 23:01:37.185369 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:01:37.185407 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:01:37.185452 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:01:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:37Z Disabled control of system clock
I20260620 23:01:37.210623 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:38903
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:38903 with env {}
W20260620 23:01:37.520835 13944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:37.521152 13944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:37.521304 13944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:37.528373 13944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:37.528465 13944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:37.528545 13944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:37.528650 13944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:37.537496 13944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:38903
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:38903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:37.538817 13944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:37.540575 13944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:37.552039 13949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:37.552138 13950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:37.552141 13952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:37.552762 13944 server_base.cc:1034] running on GCE node
I20260620 23:01:37.553483 13944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:37.554865 13944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:37.556264 13944 hybrid_clock.cc:648] HybridClock initialized: now 1781996497556205 us; error 55 us; skew 500 ppm
I20260620 23:01:37.556599 13944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:37.558794 13944 webserver.cc:460] Webserver started at http://127.10.134.190:37447/ using document root <none> and password file <none>
I20260620 23:01:37.559485 13944 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:37.559629 13944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:37.559921 13944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:37.562503 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e718ce3388774d5488cd92156d9d23be"
format_stamp: "Formatted at 2026-06-20 23:01:37 on dist-test-slave-8k25"
I20260620 23:01:37.563223 13944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e718ce3388774d5488cd92156d9d23be"
format_stamp: "Formatted at 2026-06-20 23:01:37 on dist-test-slave-8k25"
I20260620 23:01:37.567986 13944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:01:37.571151 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:37.572404 13944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:37.572571 13944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e718ce3388774d5488cd92156d9d23be"
format_stamp: "Formatted at 2026-06-20 23:01:37 on dist-test-slave-8k25"
I20260620 23:01:37.572762 13944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:37.599958 13944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:37.600754 13944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:37.601055 13944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:37.617774 13944 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:38903
I20260620 23:01:37.617802 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:38903 every 8 connection(s)
I20260620 23:01:37.619095 13944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:01:37.622658 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:37.626595 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 13944
I20260620 23:01:37.626786 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:01:37.633217 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be: Bootstrap starting.
I20260620 23:01:37.636626 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:37.637743 14010 log.cc:824] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:37.640514 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be: No bootstrap required, opened a new log
I20260620 23:01:37.648088 14010 raft_consensus.cc:348] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718ce3388774d5488cd92156d9d23be" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38903 } }
I20260620 23:01:37.648449 14010 raft_consensus.cc:374] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:37.648581 14010 raft_consensus.cc:729] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e718ce3388774d5488cd92156d9d23be, State: Initialized, Role: FOLLOWER
I20260620 23:01:37.649188 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718ce3388774d5488cd92156d9d23be" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38903 } }
I20260620 23:01:37.649420 14010 raft_consensus.cc:388] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:37.649575 14010 raft_consensus.cc:482] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:37.649747 14010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.651729 14010 raft_consensus.cc:504] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718ce3388774d5488cd92156d9d23be" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38903 } }
I20260620 23:01:37.652187 14010 leader_election.cc:302] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e718ce3388774d5488cd92156d9d23be; no voters:
I20260620 23:01:37.652777 14010 leader_election.cc:288] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:37.652895 14015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:37.653594 14015 raft_consensus.cc:686] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [term 1 LEADER]: Becoming Leader. State: Replica: e718ce3388774d5488cd92156d9d23be, State: Running, Role: LEADER
I20260620 23:01:37.654145 14015 consensus_queue.cc:237] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718ce3388774d5488cd92156d9d23be" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38903 } }
I20260620 23:01:37.654776 14010 sys_catalog.cc:564] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:37.656919 14016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e718ce3388774d5488cd92156d9d23be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718ce3388774d5488cd92156d9d23be" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38903 } } }
I20260620 23:01:37.657348 14016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:37.659261 14017 sys_catalog.cc:455] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [sys.catalog]: SysCatalogTable state changed. Reason: New leader e718ce3388774d5488cd92156d9d23be. Latest consensus state: current_term: 1 leader_uuid: "e718ce3388774d5488cd92156d9d23be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e718ce3388774d5488cd92156d9d23be" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38903 } } }
I20260620 23:01:37.659518 14017 sys_catalog.cc:458] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:37.660018 14020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:37.664826 14020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:37.674073 14020 catalog_manager.cc:1347] Generated new cluster ID: 97d041958f644e5db460e7dbab48efa9
I20260620 23:01:37.674178 14020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:37.689394 14020 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:37.690624 14020 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:37.698707 14020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e718ce3388774d5488cd92156d9d23be: Generated new TSK 0
I20260620 23:01:37.699626 14020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:37.716399 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38903
--builtin_ntp_servers=127.10.134.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260620 23:01:38.032651 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:38.032977 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:38.033126 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:38.040016 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:38.040226 14034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:01:38.048898 14034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:38903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:38.050127 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:38.051796 14034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:38.064643 14039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:38.064584 14042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:38.064692 14040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:38.065490 14034 server_base.cc:1034] running on GCE node
I20260620 23:01:38.066161 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:38.067072 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:38.068295 14034 hybrid_clock.cc:648] HybridClock initialized: now 1781996498068228 us; error 41 us; skew 500 ppm
I20260620 23:01:38.068591 14034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:38.070770 14034 webserver.cc:460] Webserver started at http://127.10.134.129:42051/ using document root <none> and password file <none>
I20260620 23:01:38.071415 14034 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:38.071573 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:38.071861 14034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:38.074376 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d1788e51ad95470ab1cbd2588102d246"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.075101 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d1788e51ad95470ab1cbd2588102d246"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.080156 14034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:01:38.083485 14048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.084832 14034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:01:38.084990 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d1788e51ad95470ab1cbd2588102d246"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.085173 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:38.098516 14034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:38.099274 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:38.099551 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:38.100395 14034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:38.101727 14034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:38.101827 14034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.102020 14034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:38.102093 14034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 23:01:38.134334 14034 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:32827
I20260620 23:01:38.134373 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:32827 every 8 connection(s)
I20260620 23:01:38.135599 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:01:38.144127 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 14034
I20260620 23:01:38.144328 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:01:38.147830 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38903
--builtin_ntp_servers=127.10.134.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260620 23:01:38.149488 14161 heartbeater.cc:347] Connected to a master server at 127.10.134.190:38903
I20260620 23:01:38.149792 14161 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:38.150683 14161 heartbeater.cc:511] Master 127.10.134.190:38903 requested a full tablet report, sending...
I20260620 23:01:38.152953 13975 ts_manager.cc:194] Registered new tserver with Master: d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:38.154585 13975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:53577
W20260620 23:01:38.461226 14164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:38.461539 14164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:38.461701 14164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:38.468760 14164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:38.468973 14164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:01:38.477730 14164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:38903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:38.479000 14164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:38.480683 14164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:38.493072 14169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:38.493201 14170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:38.493232 14172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:38.494043 14164 server_base.cc:1034] running on GCE node
I20260620 23:01:38.494606 14164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:38.495496 14164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:38.496733 14164 hybrid_clock.cc:648] HybridClock initialized: now 1781996498496662 us; error 48 us; skew 500 ppm
I20260620 23:01:38.497027 14164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:38.499233 14164 webserver.cc:460] Webserver started at http://127.10.134.130:42949/ using document root <none> and password file <none>
I20260620 23:01:38.499877 14164 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:38.500025 14164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:38.500309 14164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:38.502803 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0e509bac0552423badcb792c717c644b"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.503530 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0e509bac0552423badcb792c717c644b"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.508287 14164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:01:38.511488 14178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.512692 14164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:38.512866 14164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0e509bac0552423badcb792c717c644b"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.513069 14164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:38.538136 14164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:38.539309 14164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:38.539681 14164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:38.540714 14164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:38.542644 14164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:38.542737 14164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.542857 14164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:38.542917 14164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.575841 14164 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:39957
I20260620 23:01:38.575927 14290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:39957 every 8 connection(s)
I20260620 23:01:38.577176 14164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:01:38.587587 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 14164
I20260620 23:01:38.587733 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:01:38.591364 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38903
--builtin_ntp_servers=127.10.134.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260620 23:01:38.591652 14291 heartbeater.cc:347] Connected to a master server at 127.10.134.190:38903
I20260620 23:01:38.591967 14291 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:38.592845 14291 heartbeater.cc:511] Master 127.10.134.190:38903 requested a full tablet report, sending...
I20260620 23:01:38.594287 13975 ts_manager.cc:194] Registered new tserver with Master: 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:38.595062 13975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:42699
W20260620 23:01:38.917469 14294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:38.917750 14294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:38.917838 14294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:38.924863 14294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:38.925021 14294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:01:38.933715 14294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.10.134.190:38903
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:01:38.934942 14294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:38.936569 14294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:38.949381 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:38.949402 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:38.949606 14300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:38.950292 14294 server_base.cc:1034] running on GCE node
I20260620 23:01:38.950807 14294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:38.951632 14294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:38.952843 14294 hybrid_clock.cc:648] HybridClock initialized: now 1781996498952788 us; error 28 us; skew 500 ppm
I20260620 23:01:38.953135 14294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:01:38.955565 14294 webserver.cc:460] Webserver started at http://127.10.134.131:40163/ using document root <none> and password file <none>
I20260620 23:01:38.956238 14294 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:38.956368 14294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:38.956686 14294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:38.959227 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fbe83de2708745dda707678c226dc34f"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.959952 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fbe83de2708745dda707678c226dc34f"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.964715 14294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:01:38.967985 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.969297 14294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:01:38.969457 14294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fbe83de2708745dda707678c226dc34f"
format_stamp: "Formatted at 2026-06-20 23:01:38 on dist-test-slave-8k25"
I20260620 23:01:38.969666 14294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:38.983592 14294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:38.984454 14294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:38.984774 14294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:38.985699 14294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:38.987121 14294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:38.987191 14294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:38.987365 14294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:38.987428 14294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:39.020092 14294 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:37463
I20260620 23:01:39.020182 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:37463 every 8 connection(s)
I20260620 23:01:39.021420 14294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:01:39.031379 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 14294
I20260620 23:01:39.031564 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:01:39.034876 14421 heartbeater.cc:347] Connected to a master server at 127.10.134.190:38903
I20260620 23:01:39.035200 14421 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:39.035840 14421 heartbeater.cc:511] Master 127.10.134.190:38903 requested a full tablet report, sending...
I20260620 23:01:39.037021 13975 ts_manager.cc:194] Registered new tserver with Master: fbe83de2708745dda707678c226dc34f (127.10.134.131:37463)
I20260620 23:01:39.037729 13975 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:39473
I20260620 23:01:39.047289 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:01:39.050345 10778 test_util.cc:268] Using random seed: 1683038018
I20260620 23:01:39.068683 13975 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55546:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:39.070115 13975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:39.110164 14226 tablet_service.cc:1467] Processing CreateTablet for tablet 2644f1431b0848d98538f6b63c17382e (DEFAULT_TABLE table=test-workload [id=1e82ffd0184f476d8bf799b47a92507c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.110627 14356 tablet_service.cc:1467] Processing CreateTablet for tablet 2644f1431b0848d98538f6b63c17382e (DEFAULT_TABLE table=test-workload [id=1e82ffd0184f476d8bf799b47a92507c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.111670 14226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2644f1431b0848d98538f6b63c17382e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.112035 14356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2644f1431b0848d98538f6b63c17382e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.112251 14096 tablet_service.cc:1467] Processing CreateTablet for tablet 2644f1431b0848d98538f6b63c17382e (DEFAULT_TABLE table=test-workload [id=1e82ffd0184f476d8bf799b47a92507c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.113255 14096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2644f1431b0848d98538f6b63c17382e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.123648 14439 tablet_bootstrap.cc:492] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246: Bootstrap starting.
I20260620 23:01:39.125450 14440 tablet_bootstrap.cc:492] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b: Bootstrap starting.
I20260620 23:01:39.127457 14439 tablet_bootstrap.cc:654] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.128698 14439 log.cc:824] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:39.128850 14440 tablet_bootstrap.cc:654] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.130069 14440 log.cc:824] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:39.131119 14441 tablet_bootstrap.cc:492] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f: Bootstrap starting.
I20260620 23:01:39.131681 14439 tablet_bootstrap.cc:492] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246: No bootstrap required, opened a new log
I20260620 23:01:39.131935 14439 ts_tablet_manager.cc:1397] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 23:01:39.132517 14440 tablet_bootstrap.cc:492] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b: No bootstrap required, opened a new log
I20260620 23:01:39.132764 14440 ts_tablet_manager.cc:1397] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:01:39.134408 14441 tablet_bootstrap.cc:654] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.135475 14441 log.cc:824] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:39.138170 14441 tablet_bootstrap.cc:492] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f: No bootstrap required, opened a new log
I20260620 23:01:39.138404 14441 ts_tablet_manager.cc:1397] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:01:39.141361 14440 raft_consensus.cc:348] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.141767 14440 raft_consensus.cc:374] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.141867 14440 raft_consensus.cc:729] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e509bac0552423badcb792c717c644b, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.142494 14440 consensus_queue.cc:260] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.150190 14441 raft_consensus.cc:348] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.150671 14441 raft_consensus.cc:374] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.150789 14441 raft_consensus.cc:729] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe83de2708745dda707678c226dc34f, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.151569 14441 consensus_queue.cc:260] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.156363 14439 raft_consensus.cc:348] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.156913 14439 raft_consensus.cc:374] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.157027 14439 raft_consensus.cc:729] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1788e51ad95470ab1cbd2588102d246, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.157477 14161 heartbeater.cc:503] Master 127.10.134.190:38903 was elected leader, sending a full tablet report...
I20260620 23:01:39.159425 14439 consensus_queue.cc:260] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.160861 14449 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:39.161186 14449 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.167228 14291 heartbeater.cc:503] Master 127.10.134.190:38903 was elected leader, sending a full tablet report...
I20260620 23:01:39.167972 14421 heartbeater.cc:503] Master 127.10.134.190:38903 was elected leader, sending a full tablet report...
I20260620 23:01:39.167220 14446 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:39.169025 14446 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.170004 14448 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:39.171357 14448 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.171882 14439 ts_tablet_manager.cc:1428] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246: Time spent starting tablet: real 0.040s user 0.012s sys 0.003s
I20260620 23:01:39.185832 14446 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:39.186364 14449 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:39.190598 14440 ts_tablet_manager.cc:1428] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b: Time spent starting tablet: real 0.058s user 0.035s sys 0.024s
W20260620 23:01:39.210834 14311 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.134.129:32827 timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20260620 23:01:39.212193 14448 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
W20260620 23:01:39.212958 14051 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.131:37463 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260620 23:01:39.220494 14441 ts_tablet_manager.cc:1428] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f: Time spent starting tablet: real 0.082s user 0.015s sys 0.007s
W20260620 23:01:39.226449 14181 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.131:37463 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260620 23:01:39.227321 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:39.227797 14376 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1788e51ad95470ab1cbd2588102d246 in term 0.
I20260620 23:01:39.227898 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:39.228395 14116 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe83de2708745dda707678c226dc34f in term 0.
I20260620 23:01:39.228420 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f; no voters:
I20260620 23:01:39.229159 14449 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260620 23:01:39.229282 14050 leader_election.cc:334] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0e509bac0552423badcb792c717c644b (127.10.134.130:39957): Timed out: RequestConsensusVote RPC to 127.10.134.130:39957 timed out after 0.007s (SENT)
I20260620 23:01:39.229332 14449 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:39.229600 14449 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.230016 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:39.230378 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f; no voters:
I20260620 23:01:39.230473 14246 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1788e51ad95470ab1cbd2588102d246 in term 0.
I20260620 23:01:39.230805 14447 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:39.230967 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:39.230916 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:39.231105 14447 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.232525 14449 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.232738 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:39.233000 14116 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e509bac0552423badcb792c717c644b in current term 1: Already voted for candidate d1788e51ad95470ab1cbd2588102d246 in this term.
I20260620 23:01:39.233428 14449 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 1 election: Requested vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:39.233917 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f"
I20260620 23:01:39.234347 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b"
I20260620 23:01:39.234537 14246 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.235090 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.235777 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 1 election: Requested vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
W20260620 23:01:39.236303 14310 leader_election.cc:334] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 0e509bac0552423badcb792c717c644b (127.10.134.130:39957): Timed out: RequestConsensusVote RPC to 127.10.134.130:39957 timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20260620 23:01:39.236315 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246"
I20260620 23:01:39.236519 14116 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbe83de2708745dda707678c226dc34f in current term 1: Already voted for candidate d1788e51ad95470ab1cbd2588102d246 in this term.
I20260620 23:01:39.237432 14246 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1788e51ad95470ab1cbd2588102d246 in term 1.
I20260620 23:01:39.238134 14376 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e509bac0552423badcb792c717c644b in current term 1: Already voted for candidate fbe83de2708745dda707678c226dc34f in this term.
W20260620 23:01:39.239264 14376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.10.134.130:36469 (request call id 1) took 9 ms (client timeout 6 ms). Trace:
W20260620 23:01:39.239414 14376 rpcz_store.cc:269] 0620 23:01:39.230179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:39.230775 (+ 596us) service_pool.cc:227] Handling call
0620 23:01:39.239133 (+ 8358us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x615000082b24. stack: 00007fcc46af266e 00007fcc46af2430 00007fcc45b795ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fcc5201ad0a 00007fcc55bb45eb 00007fcc5049ee06 00007fcc5049eb35 00007fcc4d62e9be 00007fcc4d62d79e 00007fcc4d633b25 00007fcc4d63454f 00007fcc4d634378 00000000009edb7c
0620 23:01:39.239207 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15840640}
I20260620 23:01:39.240880 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246; no voters: fbe83de2708745dda707678c226dc34f
I20260620 23:01:39.241216 14449 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:39.241389 14449 raft_consensus.cc:686] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 LEADER]: Becoming Leader. State: Replica: d1788e51ad95470ab1cbd2588102d246, State: Running, Role: LEADER
W20260620 23:01:39.241418 14181 leader_election.cc:334] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer fbe83de2708745dda707678c226dc34f (127.10.134.131:37463): Timed out: RequestConsensusVote RPC to 127.10.134.131:37463 timed out after 0.006s (SENT)
I20260620 23:01:39.241559 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:39.241907 14445 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 23:01:39.241997 14449 consensus_queue.cc:237] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:39.242422 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b"
I20260620 23:01:39.242678 14246 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbe83de2708745dda707678c226dc34f in current term 1: Already voted for candidate d1788e51ad95470ab1cbd2588102d246 in this term.
I20260620 23:01:39.243675 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b"
I20260620 23:01:39.243897 14246 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbe83de2708745dda707678c226dc34f in current term 1: Already voted for candidate d1788e51ad95470ab1cbd2588102d246 in this term.
I20260620 23:01:39.244251 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:39.244488 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 23:01:39.247012 13975 catalog_manager.cc:5495] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1788e51ad95470ab1cbd2588102d246 (127.10.134.129). New cstate: current_term: 1 leader_uuid: "d1788e51ad95470ab1cbd2588102d246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:39.247589 14460 consensus_queue.cc:1035] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:01:39.298077 14463 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:01:39.298205 14463 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260620 23:01:39.629966 14246 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:01:39.629992 14375 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:01:39.630088 14375 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260620 23:01:39.630088 14246 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260620 23:01:40.025334 14445 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:40.025581 14445 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:40.026590 14445 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:40.027719 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:40.028132 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:40.028656 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:40.029155 14445 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:40.123525 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:40.123728 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:40.124697 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:40.125111 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:40.125351 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:40.126225 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:40.127337 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:40.780233 14476 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:40.780488 14476 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:40.781255 14476 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:40.781867 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:40.781919 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:40.782778 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:40.783155 14476 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:40.784255 14478 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=1, attempt_no=0}) took 1492 ms. Trace:
I20260620 23:01:40.784403 14478 rpcz_store.cc:276] 0620 23:01:39.291827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:39.291890 (+ 63us) service_pool.cc:227] Handling call
0620 23:01:40.784205 (+1492315us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:39.627075 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:39.627209 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:39.627215 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:39.627217 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:39.627312 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:39.627345 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:39.627351 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:39.627438 (+ 87us) tablet.cc:699] Row locks acquired
0620 23:01:39.627441 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:39.627498 (+ 57us) write_op.cc:270] Start()
0620 23:01:40.552660 (+925162us) write_op.cc:276] Timestamp: P: 1781996499627485 usec, L: 0
0620 23:01:40.552664 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:40.552850 (+ 186us) log.cc:842] Serialized 345 byte log entry
0620 23:01:40.782961 (+230111us) op_driver.cc:444] REPLICATION: finished
0620 23:01:40.783600 (+ 639us) write_op.cc:301] APPLY: starting
0620 23:01:40.783619 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:40.783690 (+ 71us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:40.783721 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:40.783832 (+ 111us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:40.783860 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:40.783864 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:40.783998 (+ 134us) log.cc:842] Serialized 22 byte log entry
0620 23:01:40.784085 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:40.784106 (+ 21us) write_op.cc:454] Released schema lock
0620 23:01:40.784114 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":483072,"child_traces":[["op",{"apply.queue_time_us":566,"mutex_wait_us":20,"num_ops":1,"prepare.queue_time_us":331431,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":925896,"raft.queue_time_us":309,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":489,"replication_time_us":230270,"spinlock_wait_cycles":55680,"thread_start_us":406,"threads_started":1}]]}
I20260620 23:01:41.232374 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:41.232587 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:41.233269 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:41.233819 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:41.233970 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:41.234543 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:41.481796 14476 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:41.482039 14476 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:41.482841 14476 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:41.483501 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:41.483512 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:41.484316 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:41.484846 14485 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=2, attempt_no=0}) took 1452 ms. Trace:
I20260620 23:01:41.484990 14485 rpcz_store.cc:276] 0620 23:01:40.032573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:40.032636 (+ 63us) service_pool.cc:227] Handling call
0620 23:01:41.484732 (+1452096us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:40.553011 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:40.553144 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:40.553150 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:40.553152 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:40.553291 (+ 139us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:40.553307 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:40.553315 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:40.553387 (+ 72us) tablet.cc:699] Row locks acquired
0620 23:01:40.553389 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:40.553436 (+ 47us) write_op.cc:270] Start()
0620 23:01:40.919621 (+366185us) write_op.cc:276] Timestamp: P: 1781996500553351 usec, L: 0
0620 23:01:40.919629 (+ 8us) op_driver.cc:331] REPLICATION: starting
0620 23:01:40.919881 (+ 252us) log.cc:842] Serialized 354 byte log entry
0620 23:01:41.483545 (+563664us) op_driver.cc:444] REPLICATION: finished
0620 23:01:41.484180 (+ 635us) write_op.cc:301] APPLY: starting
0620 23:01:41.484199 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:41.484245 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:41.484278 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:41.484353 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:41.484389 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:41.484396 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:41.484495 (+ 99us) log.cc:842] Serialized 22 byte log entry
0620 23:01:41.484598 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:41.484617 (+ 19us) write_op.cc:454] Released schema lock
0620 23:01:41.484625 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":573,"num_ops":1,"prepare.queue_time_us":519818,"prepare.run_cpu_time_us":857,"prepare.run_wall_time_us":366930,"replication_time_us":563886,"thread_start_us":508,"threads_started":1}]]}
I20260620 23:01:42.174989 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:42.280021 14476 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:42.282265 14492 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=3, attempt_no=0}) took 1495 ms. Trace:
I20260620 23:01:42.282433 14492 rpcz_store.cc:276] 0620 23:01:40.787106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:40.787172 (+ 66us) service_pool.cc:227] Handling call
0620 23:01:42.282211 (+1495039us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:40.919973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:40.920110 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:40.920116 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:40.920118 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:40.920226 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:40.920241 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:40.920246 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:40.920307 (+ 61us) tablet.cc:699] Row locks acquired
0620 23:01:40.920309 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:40.920365 (+ 56us) write_op.cc:270] Start()
0620 23:01:41.236512 (+316147us) write_op.cc:276] Timestamp: P: 1781996500920289 usec, L: 0
0620 23:01:41.236516 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:41.236688 (+ 172us) log.cc:842] Serialized 354 byte log entry
0620 23:01:42.281227 (+1044539us) op_driver.cc:444] REPLICATION: finished
0620 23:01:42.281686 (+ 459us) write_op.cc:301] APPLY: starting
0620 23:01:42.281705 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:42.281747 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:42.281772 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:42.281881 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:42.281909 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:42.281913 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:42.282051 (+ 138us) log.cc:842] Serialized 22 byte log entry
0620 23:01:42.282114 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:42.282129 (+ 15us) write_op.cc:454] Released schema lock
0620 23:01:42.282134 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":398,"num_ops":1,"prepare.queue_time_us":132309,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":316776,"replication_time_us":1044688,"thread_start_us":283,"threads_started":1}]]}
I20260620 23:01:43.116454 14476 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:43.116675 14476 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:43.117431 14476 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:43.118027 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:43.118005 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:43.118839 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:43.119112 14476 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:43.119539 14498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=4, attempt_no=0}) took 1632 ms. Trace:
I20260620 23:01:43.119637 14498 rpcz_store.cc:276] 0620 23:01:41.486757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:41.486826 (+ 69us) service_pool.cc:227] Handling call
0620 23:01:43.119483 (+1632657us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:41.487289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:41.487375 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:41.487378 (+ 3us) write_op.cc:435] Acquired schema lock
0620 23:01:41.487380 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:41.487482 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:41.487492 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:41.487498 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:41.487554 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:01:41.487555 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:41.487586 (+ 31us) write_op.cc:270] Start()
0620 23:01:41.989749 (+502163us) write_op.cc:276] Timestamp: P: 1781996501487510 usec, L: 0
0620 23:01:41.989754 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:41.990055 (+ 301us) log.cc:842] Serialized 354 byte log entry
0620 23:01:43.118422 (+1128367us) op_driver.cc:444] REPLICATION: finished
0620 23:01:43.118973 (+ 551us) write_op.cc:301] APPLY: starting
0620 23:01:43.118991 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:43.119034 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:43.119066 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:43.119137 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:43.119172 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:43.119178 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:43.119281 (+ 103us) log.cc:842] Serialized 22 byte log entry
0620 23:01:43.119367 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:43.119385 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:43.119392 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":466,"num_ops":1,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":766,"prepare.run_wall_time_us":502835,"replication_time_us":1128644,"thread_start_us":350,"threads_started":1}]]}
I20260620 23:01:43.326992 14476 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:43.327198 14476 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:43.327952 14476 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:43.328814 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:43.328821 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:43.329197 14498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=5, attempt_no=0}) took 1044 ms. Trace:
I20260620 23:01:43.329509 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:43.329771 14476 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:43.329380 14498 rpcz_store.cc:276] 0620 23:01:42.284357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:42.284427 (+ 70us) service_pool.cc:227] Handling call
0620 23:01:43.329131 (+1044704us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:42.284853 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:42.284943 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:42.284947 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:42.284949 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:42.285017 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:42.285025 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:42.285028 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:42.285071 (+ 43us) tablet.cc:699] Row locks acquired
0620 23:01:42.285073 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:42.285105 (+ 32us) write_op.cc:270] Start()
0620 23:01:42.661249 (+376144us) write_op.cc:276] Timestamp: P: 1781996502285060 usec, L: 0
0620 23:01:42.661254 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:42.661474 (+ 220us) log.cc:842] Serialized 354 byte log entry
0620 23:01:43.328002 (+666528us) op_driver.cc:444] REPLICATION: finished
0620 23:01:43.328369 (+ 367us) write_op.cc:301] APPLY: starting
0620 23:01:43.328387 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:43.328431 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:43.328456 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:43.328561 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:43.328601 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:43.328606 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:43.328699 (+ 93us) log.cc:842] Serialized 22 byte log entry
0620 23:01:43.328979 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:43.329001 (+ 22us) write_op.cc:454] Released schema lock
0620 23:01:43.329020 (+ 19us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":75,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":653,"prepare.run_wall_time_us":376717,"replication_time_us":666718}]]}
I20260620 23:01:43.627197 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:43.627391 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:43.627800 14375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.129:41743 (request call id 459) took 1451 ms. Trace:
I20260620 23:01:43.628214 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:43.628562 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:43.629045 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:43.627987 14375 rpcz_store.cc:276] 0620 23:01:42.176631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:42.176698 (+ 67us) service_pool.cc:227] Handling call
0620 23:01:42.176895 (+ 197us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:42.177108 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 4
0620 23:01:42.177843 (+ 735us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:42.177965 (+ 122us) write_op.cc:270] Start()
0620 23:01:42.852127 (+674162us) write_op.cc:276] Timestamp: P: 1781996500920289 usec, L: 0
0620 23:01:42.852817 (+ 690us) write_op.cc:270] Start()
0620 23:01:43.626009 (+773192us) write_op.cc:276] Timestamp: P: 1781996501487510 usec, L: 0
0620 23:01:43.626722 (+ 713us) log.cc:842] Serialized 708 byte log entry
0620 23:01:43.627066 (+ 344us) raft_consensus.cc:1631] Marking committed up to 4
0620 23:01:43.627074 (+ 8us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:01:43.627107 (+ 33us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:01:43.627708 (+ 601us) raft_consensus.cc:1671] finished
0620 23:01:43.627711 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:01:43.627756 (+ 45us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:42.852666 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:42.852815 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:42.852821 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:42.852824 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:42.852935 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:42.852948 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:42.852952 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:42.852999 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:01:42.853001 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:01:43.626552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:43.626707 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:43.626714 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:43.626716 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:43.626936 (+ 220us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:43.626989 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:43.626997 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:43.627072 (+ 75us) tablet.cc:699] Row locks acquired
0620 23:01:43.627076 (+ 4us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":264,"threads_started":1,"wal-append.queue_time_us":502,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":393,"prepare.run_cpu_time_us":443,"prepare.run_wall_time_us":444,"thread_start_us":284,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":40,"prepare.queue_time_us":412,"prepare.run_cpu_time_us":606,"prepare.run_wall_time_us":666,"thread_start_us":300,"threads_started":1}]]}
I20260620 23:01:43.629647 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:43.629916 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:44.018229 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:44.018541 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:44.019366 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:44.019870 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:44.019907 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:44.020925 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:44.021915 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:44.290499 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:44.290711 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:44.292192 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:44.291507 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:44.292964 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:44.293490 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:44.949074 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:44.949276 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:44.950151 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:44.950801 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:44.950981 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:44.951469 14511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=7, attempt_no=0}) took 1620 ms. Trace:
I20260620 23:01:44.951517 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:44.951592 14511 rpcz_store.cc:276] 0620 23:01:43.331250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:43.331324 (+ 74us) service_pool.cc:227] Handling call
0620 23:01:44.951413 (+1620089us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:43.865723 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:43.865840 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:43.865845 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:43.865846 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:43.865922 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:43.865982 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:43.865986 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:43.866044 (+ 58us) tablet.cc:699] Row locks acquired
0620 23:01:43.866046 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:43.866097 (+ 51us) write_op.cc:270] Start()
0620 23:01:44.509269 (+643172us) write_op.cc:276] Timestamp: P: 1781996503866035 usec, L: 0
0620 23:01:44.509275 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:44.509521 (+ 246us) log.cc:842] Serialized 354 byte log entry
0620 23:01:44.950225 (+440704us) op_driver.cc:444] REPLICATION: finished
0620 23:01:44.950660 (+ 435us) write_op.cc:301] APPLY: starting
0620 23:01:44.950680 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:44.950721 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:44.950779 (+ 58us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:44.950859 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:44.950894 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:44.950899 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:44.951024 (+ 125us) log.cc:842] Serialized 43 byte log entry
0620 23:01:44.951271 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:44.951288 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:44.951295 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":372,"num_ops":1,"prepare.queue_time_us":533951,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":643876,"replication_time_us":440921,"thread_start_us":262,"threads_started":1}]]}
I20260620 23:01:44.952152 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:44.965549 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:45.515535 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:45.515750 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:45.516606 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:45.517009 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:45.517275 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:45.517838 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:45.535668 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:45.535861 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:45.536725 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:45.537040 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:45.537601 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:45.538165 14519 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=8, attempt_no=0}) took 1513 ms. Trace:
I20260620 23:01:45.538339 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:45.538607 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:45.538336 14519 rpcz_store.cc:276] 0620 23:01:44.024375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:44.024442 (+ 67us) service_pool.cc:227] Handling call
0620 23:01:45.538107 (+1513665us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:44.509630 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:44.509763 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:44.509772 (+ 9us) write_op.cc:435] Acquired schema lock
0620 23:01:44.509775 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:44.509879 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:44.509889 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:44.509892 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:44.510000 (+ 108us) tablet.cc:699] Row locks acquired
0620 23:01:44.510003 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:44.510055 (+ 52us) write_op.cc:270] Start()
0620 23:01:45.231235 (+721180us) write_op.cc:276] Timestamp: P: 1781996504509984 usec, L: 0
0620 23:01:45.231240 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:45.231517 (+ 277us) log.cc:842] Serialized 354 byte log entry
0620 23:01:45.536839 (+305322us) op_driver.cc:444] REPLICATION: finished
0620 23:01:45.537475 (+ 636us) write_op.cc:301] APPLY: starting
0620 23:01:45.537494 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:45.537537 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:45.537577 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:45.537710 (+ 133us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:45.537747 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:45.537756 (+ 9us) write_op.cc:312] APPLY: finished
0620 23:01:45.537870 (+ 114us) log.cc:842] Serialized 22 byte log entry
0620 23:01:45.537988 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:45.538007 (+ 19us) write_op.cc:454] Released schema lock
0620 23:01:45.538013 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":578,"num_ops":1,"prepare.queue_time_us":484646,"prepare.run_cpu_time_us":887,"prepare.run_wall_time_us":721960,"replication_time_us":305573,"thread_start_us":259,"threads_started":1}]]}
I20260620 23:01:45.943389 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:46.094054 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:46.094455 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:46.095546 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:46.095851 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:46.096331 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:46.096524 14521 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=9, attempt_no=0}) took 1142 ms. Trace:
I20260620 23:01:46.096952 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:46.096680 14521 rpcz_store.cc:276] 0620 23:01:44.953728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:44.953806 (+ 78us) service_pool.cc:227] Handling call
0620 23:01:46.096469 (+1142663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:45.231651 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:45.231780 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:45.231785 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:45.231786 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:45.231860 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:45.231870 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:45.231873 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:45.231919 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:01:45.231920 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:45.231978 (+ 58us) write_op.cc:270] Start()
0620 23:01:45.818128 (+586150us) write_op.cc:276] Timestamp: P: 1781996505231887 usec, L: 0
0620 23:01:45.818132 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:45.818405 (+ 273us) log.cc:842] Serialized 354 byte log entry
0620 23:01:46.095092 (+276687us) op_driver.cc:444] REPLICATION: finished
0620 23:01:46.095869 (+ 777us) write_op.cc:301] APPLY: starting
0620 23:01:46.095887 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:46.095927 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:46.095976 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:46.096082 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:46.096122 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:46.096129 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:46.096261 (+ 132us) log.cc:842] Serialized 43 byte log entry
0620 23:01:46.096329 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:46.096347 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:46.096354 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":704,"num_ops":1,"prepare.queue_time_us":277315,"prepare.run_cpu_time_us":768,"prepare.run_wall_time_us":586824,"replication_time_us":276931,"thread_start_us":1541,"threads_started":1}]]}
I20260620 23:01:46.097254 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:46.792156 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:46.792357 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:46.793136 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:46.793732 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:46.793735 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:46.794426 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:47.029160 14447 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:47.029356 14447 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:47.030155 14447 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:47.030627 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:47.030865 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:47.032083 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:47.036338 14447 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:47.036985 14528 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=10, attempt_no=0}) took 1496 ms. Trace:
I20260620 23:01:47.037138 14528 rpcz_store.cc:276] 0620 23:01:45.540326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:45.540400 (+ 74us) service_pool.cc:227] Handling call
0620 23:01:47.036927 (+1496527us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:45.818510 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:45.818648 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:45.818656 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:45.818659 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:45.818759 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:45.818772 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:45.818776 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:45.818835 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:01:45.818838 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:45.818915 (+ 77us) write_op.cc:270] Start()
0620 23:01:46.411087 (+592172us) write_op.cc:276] Timestamp: P: 1781996505818877 usec, L: 0
0620 23:01:46.411094 (+ 7us) op_driver.cc:331] REPLICATION: starting
0620 23:01:46.411327 (+ 233us) log.cc:842] Serialized 354 byte log entry
0620 23:01:47.030997 (+619670us) op_driver.cc:444] REPLICATION: finished
0620 23:01:47.035890 (+ 4893us) write_op.cc:301] APPLY: starting
0620 23:01:47.035915 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:47.035969 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:47.035999 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:47.036112 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:47.036156 (+ 44us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:47.036166 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:01:47.036263 (+ 97us) log.cc:842] Serialized 22 byte log entry
0620 23:01:47.036344 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:47.036561 (+ 217us) write_op.cc:454] Released schema lock
0620 23:01:47.036571 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4823,"num_ops":1,"prepare.queue_time_us":277691,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":592889,"replication_time_us":619877,"thread_start_us":4708,"threads_started":1}]]}
I20260620 23:01:47.231336 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:47.741775 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:47.742007 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:47.742748 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:47.743237 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:47.743759 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:47.744310 14533 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=11, attempt_no=0}) took 1645 ms. Trace:
I20260620 23:01:47.744323 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:47.744602 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:47.744501 14533 rpcz_store.cc:276] 0620 23:01:46.098925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:46.098994 (+ 69us) service_pool.cc:227] Handling call
0620 23:01:47.744256 (+1645262us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:46.411448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:46.411580 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:46.411587 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:46.411589 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:46.411692 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:46.411703 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:46.411710 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:47.036391 (+624681us) lock_manager.cc:599] Waited 624521us for lock on <redacted> (key hash 16040139233676677207)
0620 23:01:47.036408 (+ 17us) tablet.cc:699] Row locks acquired
0620 23:01:47.036411 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:47.036457 (+ 46us) write_op.cc:270] Start()
0620 23:01:47.545591 (+509134us) write_op.cc:276] Timestamp: P: 1781996507036408 usec, L: 0
0620 23:01:47.545597 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:47.545828 (+ 231us) log.cc:842] Serialized 354 byte log entry
0620 23:01:47.742850 (+197022us) op_driver.cc:444] REPLICATION: finished
0620 23:01:47.743307 (+ 457us) write_op.cc:301] APPLY: starting
0620 23:01:47.743358 (+ 51us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:47.743422 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:47.743425 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:47.743569 (+ 144us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:47.743607 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:47.743615 (+ 8us) write_op.cc:312] APPLY: finished
0620 23:01:47.743768 (+ 153us) log.cc:842] Serialized 43 byte log entry
0620 23:01:47.744010 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:47.744026 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:47.744101 (+ 75us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":396,"num_ops":1,"prepare.queue_time_us":312037,"prepare.run_cpu_time_us":1296,"prepare.run_wall_time_us":1134845,"raft.queue_time_us":168,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":615,"replication_time_us":197218,"row_lock_wait_count":1,"row_lock_wait_us":624521,"spinlock_wait_cycles":1347328,"thread_start_us":570,"threads_started":2}]]}
I20260620 23:01:48.437004 14531 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:48.437319 14531 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:48.438207 14531 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:48.438589 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:48.438783 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:48.439420 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:48.542816 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:48.543040 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:48.543792 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:48.544346 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:48.544450 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:48.545132 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:48.545461 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:48.547482 14541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=12, attempt_no=0}) took 1508 ms. Trace:
I20260620 23:01:48.547590 14541 rpcz_store.cc:276] 0620 23:01:47.038960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:47.039020 (+ 60us) service_pool.cc:227] Handling call
0620 23:01:48.547431 (+1508411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:47.546321 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:47.546447 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:47.546454 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:47.546456 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:47.546550 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:47.546560 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:47.546564 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:47.546634 (+ 70us) tablet.cc:699] Row locks acquired
0620 23:01:47.546637 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:47.546685 (+ 48us) write_op.cc:270] Start()
0620 23:01:47.902859 (+356174us) write_op.cc:276] Timestamp: P: 1781996507546619 usec, L: 0
0620 23:01:47.902866 (+ 7us) op_driver.cc:331] REPLICATION: starting
0620 23:01:47.903130 (+ 264us) log.cc:842] Serialized 354 byte log entry
0620 23:01:48.546467 (+643337us) op_driver.cc:444] REPLICATION: finished
0620 23:01:48.546912 (+ 445us) write_op.cc:301] APPLY: starting
0620 23:01:48.546931 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:48.546971 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:48.547001 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:48.547090 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:48.547125 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:48.547132 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:48.547220 (+ 88us) log.cc:842] Serialized 22 byte log entry
0620 23:01:48.547312 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:48.547332 (+ 20us) write_op.cc:454] Released schema lock
0620 23:01:48.547340 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":376,"num_ops":1,"prepare.queue_time_us":506956,"prepare.run_cpu_time_us":812,"prepare.run_wall_time_us":356888,"replication_time_us":643574,"thread_start_us":272,"threads_started":1}]]}
I20260620 23:01:48.987530 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:48.987730 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:48.988487 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:48.989006 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:48.989053 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:48.989668 14541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=13, attempt_no=0}) took 1243 ms. Trace:
I20260620 23:01:48.989784 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:48.989787 14541 rpcz_store.cc:276] 0620 23:01:47.746502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:47.746567 (+ 65us) service_pool.cc:227] Handling call
0620 23:01:48.989608 (+1243041us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:47.903261 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:47.903392 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:47.903399 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:47.903401 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:47.903501 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:47.903513 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:47.903517 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:47.903574 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:01:47.903577 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:47.903625 (+ 48us) write_op.cc:270] Start()
0620 23:01:48.123797 (+220172us) write_op.cc:276] Timestamp: P: 1781996507903578 usec, L: 0
0620 23:01:48.123802 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:48.124185 (+ 383us) log.cc:842] Serialized 354 byte log entry
0620 23:01:48.988810 (+864625us) op_driver.cc:444] REPLICATION: finished
0620 23:01:48.989064 (+ 254us) write_op.cc:301] APPLY: starting
0620 23:01:48.989083 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:48.989145 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:48.989195 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:48.989277 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:48.989312 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:48.989317 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:48.989402 (+ 85us) log.cc:842] Serialized 22 byte log entry
0620 23:01:48.989490 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:48.989506 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:48.989513 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":183,"num_ops":1,"prepare.queue_time_us":156208,"prepare.run_cpu_time_us":929,"prepare.run_wall_time_us":221008,"replication_time_us":864977}]]}
I20260620 23:01:48.990278 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:49.207351 14531 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:49.624718 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:49.624944 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:49.625787 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:49.626384 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:49.626436 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:49.627118 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:49.627421 14547 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=14, attempt_no=0}) took 1078 ms. Trace:
I20260620 23:01:49.627653 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:49.627524 14547 rpcz_store.cc:276] 0620 23:01:48.549330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:48.549404 (+ 74us) service_pool.cc:227] Handling call
0620 23:01:49.627301 (+1077897us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:48.549869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:48.550016 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:48.550021 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:48.550024 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:48.550120 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:48.550128 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:48.550131 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:48.550168 (+ 37us) tablet.cc:699] Row locks acquired
0620 23:01:48.550170 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:48.550200 (+ 30us) write_op.cc:270] Start()
0620 23:01:49.314344 (+764144us) write_op.cc:276] Timestamp: P: 1781996508550149 usec, L: 0
0620 23:01:49.314349 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:49.314534 (+ 185us) log.cc:842] Serialized 354 byte log entry
0620 23:01:49.625847 (+311313us) op_driver.cc:444] REPLICATION: finished
0620 23:01:49.626610 (+ 763us) write_op.cc:301] APPLY: starting
0620 23:01:49.626626 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:49.626666 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:49.626712 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:49.626792 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:49.626827 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:49.626833 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:49.627071 (+ 238us) log.cc:842] Serialized 43 byte log entry
0620 23:01:49.627158 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:49.627175 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:49.627182 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":701,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":764796,"raft.queue_time_us":125,"raft.run_cpu_time_us":597,"raft.run_wall_time_us":597,"replication_time_us":311464,"spinlock_wait_cycles":25088,"thread_start_us":1056,"threads_started":1}]]}
I20260620 23:01:49.676498 14531 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:49.676702 14531 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:49.677484 14531 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:49.678117 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:49.678112 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:49.678755 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:49.679033 14531 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:50.344606 14531 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:50.344834 14531 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:50.345585 14531 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:50.346114 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:50.346554 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:50.346907 14550 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=15, attempt_no=0}) took 1355 ms. Trace:
I20260620 23:01:50.347271 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:50.347069 14550 rpcz_store.cc:276] 0620 23:01:48.991687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:48.991766 (+ 79us) service_pool.cc:227] Handling call
0620 23:01:50.346833 (+1355067us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:49.314715 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:49.314850 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:49.314863 (+ 13us) write_op.cc:435] Acquired schema lock
0620 23:01:49.314865 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:49.314968 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:49.314983 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:49.314987 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:49.315067 (+ 80us) tablet.cc:699] Row locks acquired
0620 23:01:49.315071 (+ 4us) write_op.cc:260] PREPARE: finished
0620 23:01:49.315119 (+ 48us) write_op.cc:270] Start()
0620 23:01:50.056261 (+741142us) write_op.cc:276] Timestamp: P: 1781996509315036 usec, L: 0
0620 23:01:50.056264 (+ 3us) op_driver.cc:331] REPLICATION: starting
0620 23:01:50.056450 (+ 186us) log.cc:842] Serialized 354 byte log entry
0620 23:01:50.345671 (+289221us) op_driver.cc:444] REPLICATION: finished
0620 23:01:50.346209 (+ 538us) write_op.cc:301] APPLY: starting
0620 23:01:50.346225 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:50.346268 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:50.346348 (+ 80us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:50.346425 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:50.346460 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:50.346466 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:01:50.346602 (+ 136us) log.cc:842] Serialized 22 byte log entry
0620 23:01:50.346712 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:50.346729 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:50.346736 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":474,"num_ops":1,"prepare.queue_time_us":322488,"prepare.run_cpu_time_us":784,"prepare.run_wall_time_us":741854,"raft.queue_time_us":124,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":600,"replication_time_us":289331,"spinlock_wait_cycles":3968,"thread_start_us":326,"threads_started":1}]]}
I20260620 23:01:50.390717 14504 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:50.390954 14504 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:50.391732 14504 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:50.392258 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:50.392263 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:50.392918 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:50.853328 14531 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:50.856758 14556 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=16, attempt_no=0}) took 1227 ms. Trace:
I20260620 23:01:50.856916 14556 rpcz_store.cc:276] 0620 23:01:49.629657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:49.629709 (+ 52us) service_pool.cc:227] Handling call
0620 23:01:50.856709 (+1227000us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:50.056637 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:50.056779 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:50.056785 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:50.056787 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:50.056904 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:50.056915 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:50.056920 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:50.056977 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:01:50.056980 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:50.057028 (+ 48us) write_op.cc:270] Start()
0620 23:01:50.314193 (+257165us) write_op.cc:276] Timestamp: P: 1781996510056973 usec, L: 0
0620 23:01:50.314201 (+ 8us) op_driver.cc:331] REPLICATION: starting
0620 23:01:50.314434 (+ 233us) log.cc:842] Serialized 354 byte log entry
0620 23:01:50.855768 (+541334us) op_driver.cc:444] REPLICATION: finished
0620 23:01:50.856128 (+ 360us) write_op.cc:301] APPLY: starting
0620 23:01:50.856158 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:50.856199 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:50.856229 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:50.856349 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:50.856382 (+ 33us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:50.856390 (+ 8us) write_op.cc:312] APPLY: finished
0620 23:01:50.856486 (+ 96us) log.cc:842] Serialized 22 byte log entry
0620 23:01:50.856557 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:50.856571 (+ 14us) write_op.cc:454] Released schema lock
0620 23:01:50.856576 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":283,"num_ops":1,"prepare.queue_time_us":426432,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":257892,"replication_time_us":541539,"thread_start_us":207,"threads_started":1}]]}
I20260620 23:01:51.135185 14504 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:51.634291 14531 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:51.634518 14531 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:51.635325 14531 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:51.635739 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:51.636394 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:51.637287 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:51.637589 14531 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:52.402254 14531 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:52.402482 14531 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:52.403434 14531 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:52.403921 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:52.404050 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:52.404722 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:52.405010 14531 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:52.405613 14566 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=18, attempt_no=0}) took 1546 ms. Trace:
I20260620 23:01:52.405763 14566 rpcz_store.cc:276] 0620 23:01:50.858708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:50.858775 (+ 67us) service_pool.cc:227] Handling call
0620 23:01:52.405558 (+1546783us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:50.859206 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:50.859303 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:50.859307 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:01:50.859309 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:50.859374 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:50.859382 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:50.859385 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:50.859421 (+ 36us) tablet.cc:699] Row locks acquired
0620 23:01:50.859422 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:01:50.859451 (+ 29us) write_op.cc:270] Start()
0620 23:01:51.686570 (+827119us) write_op.cc:276] Timestamp: P: 1781996510859372 usec, L: 0
0620 23:01:51.686573 (+ 3us) op_driver.cc:331] REPLICATION: starting
0620 23:01:51.686751 (+ 178us) log.cc:842] Serialized 354 byte log entry
0620 23:01:52.403961 (+717210us) op_driver.cc:444] REPLICATION: finished
0620 23:01:52.404683 (+ 722us) write_op.cc:301] APPLY: starting
0620 23:01:52.404703 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:52.404751 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:52.404816 (+ 65us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:52.404934 (+ 118us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:52.404970 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:52.404977 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:52.405098 (+ 121us) log.cc:842] Serialized 43 byte log entry
0620 23:01:52.405389 (+ 291us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:52.405407 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:52.405416 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":655,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":827675,"raft.queue_time_us":124,"raft.run_cpu_time_us":514,"raft.run_wall_time_us":613,"replication_time_us":717363,"spinlock_wait_cycles":93696,"thread_start_us":318,"threads_started":1}]]}
I20260620 23:01:52.491192 14562 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:52.491420 14562 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:52.492151 14562 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:52.492712 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:52.492830 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:52.493897 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:52.554502 14562 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:52.556383 14566 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=19, attempt_no=0}) took 1414 ms. Trace:
I20260620 23:01:52.556531 14566 rpcz_store.cc:276] 0620 23:01:51.142310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:51.142381 (+ 71us) service_pool.cc:227] Handling call
0620 23:01:52.556323 (+1413942us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:51.686923 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:51.687052 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:51.687060 (+ 8us) write_op.cc:435] Acquired schema lock
0620 23:01:51.687062 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:51.687168 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:51.687181 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:51.687189 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:51.687245 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:01:51.687247 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:51.687300 (+ 53us) write_op.cc:270] Start()
0620 23:01:52.470450 (+783150us) write_op.cc:276] Timestamp: P: 1781996511687238 usec, L: 0
0620 23:01:52.470455 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:52.470657 (+ 202us) log.cc:842] Serialized 354 byte log entry
0620 23:01:52.555554 (+ 84897us) op_driver.cc:444] REPLICATION: finished
0620 23:01:52.555764 (+ 210us) write_op.cc:301] APPLY: starting
0620 23:01:52.555783 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:52.555828 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:52.555874 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:52.555947 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:52.555985 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:52.555990 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:52.556114 (+ 124us) log.cc:842] Serialized 43 byte log entry
0620 23:01:52.556183 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:52.556200 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:52.556207 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":138,"num_ops":1,"prepare.queue_time_us":544123,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":783834,"replication_time_us":85071}]]}
I20260620 23:01:52.807649 14531 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:52.807854 14531 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:52.808657 14531 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:52.809173 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:52.809499 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:52.810086 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:52.810351 14531 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:53.451740 14572 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:53.452071 14572 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:53.452898 14572 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:53.453451 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:53.453511 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:53.454208 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:53.454584 14572 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:53.454844 14575 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=20, attempt_no=0}) took 1046 ms. Trace:
I20260620 23:01:53.454994 14575 rpcz_store.cc:276] 0620 23:01:52.408433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:52.408518 (+ 85us) service_pool.cc:227] Handling call
0620 23:01:53.454787 (+1046269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:52.470796 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:52.470931 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:52.470937 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:52.470939 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:52.471064 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:52.471076 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:52.471081 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:52.471140 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:01:52.471142 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:52.471201 (+ 59us) write_op.cc:270] Start()
0620 23:01:53.382352 (+911151us) write_op.cc:276] Timestamp: P: 1781996512471167 usec, L: 0
0620 23:01:53.382358 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:53.382638 (+ 280us) log.cc:842] Serialized 354 byte log entry
0620 23:01:53.452832 (+ 70194us) op_driver.cc:444] REPLICATION: finished
0620 23:01:53.453369 (+ 537us) write_op.cc:301] APPLY: starting
0620 23:01:53.453386 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:53.453431 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:53.453464 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:53.454139 (+ 675us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:53.454189 (+ 50us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:53.454200 (+ 11us) write_op.cc:312] APPLY: finished
0620 23:01:53.454374 (+ 174us) log.cc:842] Serialized 43 byte log entry
0620 23:01:53.454531 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:53.454550 (+ 19us) write_op.cc:454] Released schema lock
0620 23:01:53.454558 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":478,"num_ops":1,"prepare.queue_time_us":61933,"prepare.run_cpu_time_us":897,"prepare.run_wall_time_us":911950,"raft.queue_time_us":44,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":424,"replication_time_us":70441,"spinlock_wait_cycles":61440,"thread_start_us":342,"threads_started":1}]]}
I20260620 23:01:54.143111 14573 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:54.143384 14573 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:54.144065 14573 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:54.144618 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:54.144866 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:54.145397 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:54.145677 14573 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:54.155517 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:54.155743 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:54.156591 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:54.156821 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:54.156873 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:54.157383 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:54.157606 14581 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:54.943769 14573 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:54.943970 14573 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:54.945017 14573 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:54.945626 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:54.945703 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:54.946391 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:54.946652 14573 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:54.947706 14586 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=21, attempt_no=0}) took 2389 ms. Trace:
I20260620 23:01:54.947830 14586 rpcz_store.cc:276] 0620 23:01:52.558525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:52.558637 (+ 112us) service_pool.cc:227] Handling call
0620 23:01:54.947654 (+2389017us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:53.382789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:53.382924 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:53.382930 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:53.382933 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:53.383068 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:53.383080 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:53.383085 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:53.383155 (+ 70us) tablet.cc:699] Row locks acquired
0620 23:01:53.383159 (+ 4us) write_op.cc:260] PREPARE: finished
0620 23:01:53.383209 (+ 50us) write_op.cc:270] Start()
0620 23:01:54.246358 (+863149us) write_op.cc:276] Timestamp: P: 1781996513383127 usec, L: 0
0620 23:01:54.246362 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:01:54.246605 (+ 243us) log.cc:842] Serialized 354 byte log entry
0620 23:01:54.946517 (+699912us) op_driver.cc:444] REPLICATION: finished
0620 23:01:54.947007 (+ 490us) write_op.cc:301] APPLY: starting
0620 23:01:54.947034 (+ 27us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:54.947066 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:54.947161 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:54.947242 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:54.947266 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:54.947270 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:54.947414 (+ 144us) log.cc:842] Serialized 43 byte log entry
0620 23:01:54.947484 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:54.947503 (+ 19us) write_op.cc:454] Released schema lock
0620 23:01:54.947510 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":422,"num_ops":1,"prepare.queue_time_us":823693,"prepare.run_cpu_time_us":1528,"prepare.run_wall_time_us":864603,"raft.queue_time_us":471,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":700129,"thread_start_us":967,"threads_started":2}]]}
I20260620 23:01:55.167557 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:55.167810 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:55.168823 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:55.169422 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:55.169540 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:55.170326 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:55.600185 14573 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:55.600382 14573 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:55.601147 14573 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:55.602155 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:55.602725 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:55.602774 14592 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=22, attempt_no=0}) took 2145 ms. Trace:
I20260620 23:01:55.602926 14592 rpcz_store.cc:276] 0620 23:01:53.456804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:53.456893 (+ 89us) service_pool.cc:227] Handling call
0620 23:01:55.602695 (+2145802us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:54.247435 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:54.247560 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:54.247565 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:54.247567 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:54.247667 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:54.247678 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:54.247682 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:54.247736 (+ 54us) tablet.cc:699] Row locks acquired
0620 23:01:54.247739 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:01:54.247787 (+ 48us) write_op.cc:270] Start()
0620 23:01:54.567940 (+320153us) write_op.cc:276] Timestamp: P: 1781996514247725 usec, L: 0
0620 23:01:54.567946 (+ 6us) op_driver.cc:331] REPLICATION: starting
0620 23:01:54.568199 (+ 253us) log.cc:842] Serialized 354 byte log entry
0620 23:01:55.601329 (+1033130us) op_driver.cc:444] REPLICATION: finished
0620 23:01:55.601926 (+ 597us) write_op.cc:301] APPLY: starting
0620 23:01:55.601965 (+ 39us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:55.602006 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:55.602035 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:55.602135 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:55.602172 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:55.602182 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:01:55.602267 (+ 85us) log.cc:842] Serialized 22 byte log entry
0620 23:01:55.602559 (+ 292us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:55.602576 (+ 17us) write_op.cc:454] Released schema lock
0620 23:01:55.602585 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":529,"num_ops":1,"prepare.queue_time_us":790027,"prepare.run_cpu_time_us":838,"prepare.run_wall_time_us":320900,"replication_time_us":1033348,"threads_started":1}]]}
I20260620 23:01:55.603492 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:55.683352 14581 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:56.429656 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:56.429862 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:56.430740 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:56.431241 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:56.431536 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:56.432183 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:56.432461 14581 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:56.432793 14599 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=23, attempt_no=0}) took 1482 ms. Trace:
I20260620 23:01:56.432896 14599 rpcz_store.cc:276] 0620 23:01:54.949805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:54.949873 (+ 68us) service_pool.cc:227] Handling call
0620 23:01:56.432742 (+1482869us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:54.950392 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:54.950481 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:54.950486 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:54.950487 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:54.950550 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:54.950558 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:54.950561 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:54.950607 (+ 46us) tablet.cc:699] Row locks acquired
0620 23:01:54.950609 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:54.950639 (+ 30us) write_op.cc:270] Start()
0620 23:01:55.378315 (+427676us) write_op.cc:276] Timestamp: P: 1781996514950570 usec, L: 0
0620 23:01:55.378324 (+ 9us) op_driver.cc:331] REPLICATION: starting
0620 23:01:55.378637 (+ 313us) log.cc:842] Serialized 354 byte log entry
0620 23:01:56.431640 (+1053003us) op_driver.cc:444] REPLICATION: finished
0620 23:01:56.432160 (+ 520us) write_op.cc:301] APPLY: starting
0620 23:01:56.432182 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:56.432225 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:56.432257 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:56.432351 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:56.432387 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:56.432397 (+ 10us) write_op.cc:312] APPLY: finished
0620 23:01:56.432511 (+ 114us) log.cc:842] Serialized 43 byte log entry
0620 23:01:56.432579 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:56.432597 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:56.432604 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":433,"num_ops":1,"prepare.queue_time_us":72,"prepare.run_cpu_time_us":1382,"prepare.run_wall_time_us":428335,"replication_time_us":1053288,"thread_start_us":406,"threads_started":1}]]}
I20260620 23:01:56.497081 14573 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:56.608390 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:56.608596 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:56.609417 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:56.610029 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:56.610154 14599 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=24, attempt_no=0}) took 1004 ms. Trace:
I20260620 23:01:56.610601 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:56.610278 14599 rpcz_store.cc:276] 0620 23:01:55.605179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:55.605259 (+ 80us) service_pool.cc:227] Handling call
0620 23:01:56.610083 (+1004824us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:55.605888 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:55.606152 (+ 264us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:55.606157 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:55.606158 (+ 1us) tablet.cc:660] Decoding operations
0620 23:01:55.606245 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:55.606252 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:55.606259 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:55.606350 (+ 91us) tablet.cc:699] Row locks acquired
0620 23:01:55.606352 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:55.606393 (+ 41us) write_op.cc:270] Start()
0620 23:01:56.433537 (+827144us) write_op.cc:276] Timestamp: P: 1781996515606286 usec, L: 0
0620 23:01:56.433542 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:56.433729 (+ 187us) log.cc:842] Serialized 354 byte log entry
0620 23:01:56.609408 (+175679us) op_driver.cc:444] REPLICATION: finished
0620 23:01:56.609530 (+ 122us) write_op.cc:301] APPLY: starting
0620 23:01:56.609548 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:56.609592 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:56.609621 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:56.609697 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:56.609746 (+ 49us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:56.609753 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:01:56.609840 (+ 87us) log.cc:842] Serialized 22 byte log entry
0620 23:01:56.609924 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:56.609980 (+ 56us) write_op.cc:454] Released schema lock
0620 23:01:56.609989 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":827929,"raft.queue_time_us":29,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":175836,"spinlock_wait_cycles":5888}]]}
I20260620 23:01:56.611195 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:56.611454 14581 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:57.158535 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:57.158754 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.159561 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.160020 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:57.160035 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:57.160722 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:57.161079 14581 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:57.477516 14573 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:57.477716 14573 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.478475 14573 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:57.478826 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:57.478950 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:57.479646 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:57.685575 14573 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:57.841897 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:57.842190 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.842965 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.843149 14573 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:57.843328 14573 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.843729 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:57.843986 14245 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe83de2708745dda707678c226dc34f in term 1.
I20260620 23:01:57.844017 14573 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:57.844554 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:57.845011 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f; no voters:
I20260620 23:01:57.845268 14581 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:01:57.845376 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Starting leader election (detected failure of leader d1788e51ad95470ab1cbd2588102d246)
I20260620 23:01:57.845350 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:57.845460 14581 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:57.845590 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:57.845144 14611 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47558 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=26, attempt_no=0}) took 1233 ms. Trace:
I20260620 23:01:57.846561 14611 rpcz_store.cc:276] 0620 23:01:56.612048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:56.612107 (+ 59us) service_pool.cc:227] Handling call
0620 23:01:57.845094 (+1232987us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:56.624692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:56.624813 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:56.624819 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:01:56.624821 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:56.624913 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:56.624930 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:56.624934 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:56.624996 (+ 62us) tablet.cc:699] Row locks acquired
0620 23:01:56.624998 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:56.625039 (+ 41us) write_op.cc:270] Start()
0620 23:01:57.504188 (+879149us) write_op.cc:276] Timestamp: P: 1781996516624989 usec, L: 0
0620 23:01:57.504227 (+ 39us) op_driver.cc:331] REPLICATION: starting
0620 23:01:57.504449 (+ 222us) log.cc:842] Serialized 354 byte log entry
0620 23:01:57.843405 (+338956us) op_driver.cc:444] REPLICATION: finished
0620 23:01:57.844416 (+ 1011us) write_op.cc:301] APPLY: starting
0620 23:01:57.844432 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:57.844475 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:57.844499 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:57.844580 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:57.844612 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:57.844617 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:01:57.844715 (+ 98us) log.cc:842] Serialized 43 byte log entry
0620 23:01:57.844953 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:57.844969 (+ 16us) write_op.cc:454] Released schema lock
0620 23:01:57.844975 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":950,"num_ops":1,"prepare.queue_time_us":12049,"prepare.run_cpu_time_us":771,"prepare.run_wall_time_us":879842,"replication_time_us":339152,"spinlock_wait_cycles":15232,"thread_start_us":860,"threads_started":1}]]}
I20260620 23:01:57.848124 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.848498 14376 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e509bac0552423badcb792c717c644b in current term 2: Already voted for candidate fbe83de2708745dda707678c226dc34f in this term.
I20260620 23:01:57.848742 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 election: Requested vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.849085 14375 raft_consensus.cc:1226] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Rejecting Update request from peer d1788e51ad95470ab1cbd2588102d246 for earlier term 1. Current term is 2. Ops: []
I20260620 23:01:57.849123 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b"
I20260620 23:01:57.849267 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:57.849509 14573 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:57.850464 14604 consensus_queue.cc:1046] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 }, Status: INVALID_TERM, Last received: 1.28, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20260620 23:01:57.850644 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246"
I20260620 23:01:57.851073 14605 raft_consensus.cc:3032] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:01:57.851100 14311 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe83de2708745dda707678c226dc34f; no voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:57.851181 14605 raft_consensus.cc:729] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1788e51ad95470ab1cbd2588102d246, State: Running, Role: LEADER
I20260620 23:01:57.851490 14581 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:01:57.851490 14605 consensus_queue.cc:260] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.852030 14605 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:57.852272 14573 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:57.858126 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:57.858287 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.858413 14605 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:57.858533 14605 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.858946 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.859328 14605 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.859272 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:57.859406 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:57.859475 14246 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe83de2708745dda707678c226dc34f in term 2.
I20260620 23:01:57.859647 14116 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe83de2708745dda707678c226dc34f in term 2.
I20260620 23:01:57.859793 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:57.859951 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:57.859987 14375 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1788e51ad95470ab1cbd2588102d246 in term 2.
I20260620 23:01:57.860009 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f; no voters:
I20260620 23:01:57.860150 14246 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1788e51ad95470ab1cbd2588102d246 in term 2.
I20260620 23:01:57.860338 14581 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:01:57.860483 14581 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:57.860508 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e509bac0552423badcb792c717c644b, d1788e51ad95470ab1cbd2588102d246; no voters:
I20260620 23:01:57.860625 14581 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:01:57.860764 14605 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:01:57.860878 14605 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:57.860975 14605 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:01:57.862517 14581 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.862771 14605 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.863056 14581 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 3 election: Requested vote from peers d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.863389 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246"
I20260620 23:01:57.863435 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "fbe83de2708745dda707678c226dc34f" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b"
I20260620 23:01:57.863557 14605 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 3 election: Requested vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.863606 14246 raft_consensus.cc:3037] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:01:57.863652 14116 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fbe83de2708745dda707678c226dc34f in current term 3: Already voted for candidate d1788e51ad95470ab1cbd2588102d246 in this term.
I20260620 23:01:57.863868 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f"
I20260620 23:01:57.864071 14375 raft_consensus.cc:2379] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d1788e51ad95470ab1cbd2588102d246 in current term 3: Already voted for candidate fbe83de2708745dda707678c226dc34f in this term.
I20260620 23:01:57.864208 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 3 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b"
I20260620 23:01:57.864748 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:01:57.865082 14605 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260620 23:01:57.866070 14246 raft_consensus.cc:2454] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbe83de2708745dda707678c226dc34f in term 3.
I20260620 23:01:57.866333 14310 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f; no voters: d1788e51ad95470ab1cbd2588102d246
I20260620 23:01:57.866586 14581 raft_consensus.cc:2781] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:01:57.867043 14581 raft_consensus.cc:686] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [term 3 LEADER]: Becoming Leader. State: Replica: fbe83de2708745dda707678c226dc34f, State: Running, Role: LEADER
I20260620 23:01:57.867488 14581 consensus_queue.cc:237] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.870385 14246 raft_consensus.cc:1261] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Refusing update from remote peer fbe83de2708745dda707678c226dc34f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20260620 23:01:57.870437 13974 catalog_manager.cc:5495] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f reported cstate change: term changed from 1 to 3, leader changed from d1788e51ad95470ab1cbd2588102d246 (127.10.134.129) to fbe83de2708745dda707678c226dc34f (127.10.134.131). New cstate: current_term: 3 leader_uuid: "fbe83de2708745dda707678c226dc34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:57.871025 14612 consensus_queue.cc:1035] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260620 23:01:57.871181 14116 raft_consensus.cc:1261] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Refusing update from remote peer fbe83de2708745dda707678c226dc34f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20260620 23:01:57.871881 14612 consensus_queue.cc:1035] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260620 23:01:57.873201 14604 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:01:57.873389 14604 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.873976 14604 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:57.875490 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:57.876209 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:57.876852 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:01:57.877303 14604 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:57.878252 14573 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:01:57.878439 14573 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:57.879169 14573 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:57.879837 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:57.880059 14116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:57.880592 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:01:57.881608 14573 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:58.028010 14605 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:01:58.028214 14605 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:58.028760 14605 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:58.029232 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:58.034833 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:58.035431 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:01:58.035751 14605 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260620 23:01:58.095710 14463 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47558: Illegal state: Replica d1788e51ad95470ab1cbd2588102d246 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "fbe83de2708745dda707678c226dc34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } } }
W20260620 23:01:58.096925 14463 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47558: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.000s ago.
I20260620 23:01:58.583783 14624 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:01:58.584064 14624 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:58.584980 14624 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:58.585238 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:58.586474 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:58.587003 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:01:58.587457 14624 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:59.708169 14627 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:01:59.708467 14627 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:59.709381 14627 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:01:59.709981 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:01:59.710203 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:59.710747 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:01:59.711021 14627 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:01:59.712314 14632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=28, attempt_no=1}) took 1602 ms. Trace:
I20260620 23:01:59.712411 14632 rpcz_store.cc:276] 0620 23:01:58.109803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:58.109871 (+ 68us) service_pool.cc:227] Handling call
0620 23:01:59.712270 (+1602399us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:58.111307 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:58.111436 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:58.111441 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:01:58.111445 (+ 4us) tablet.cc:660] Decoding operations
0620 23:01:58.111536 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:58.111554 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:58.111557 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:58.111601 (+ 44us) tablet.cc:699] Row locks acquired
0620 23:01:58.111603 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:58.111649 (+ 46us) write_op.cc:270] Start()
0620 23:01:59.104807 (+993158us) write_op.cc:276] Timestamp: P: 1781996518111600 usec, L: 0
0620 23:01:59.104814 (+ 7us) op_driver.cc:331] REPLICATION: starting
0620 23:01:59.105035 (+ 221us) log.cc:842] Serialized 354 byte log entry
0620 23:01:59.710864 (+605829us) op_driver.cc:444] REPLICATION: finished
0620 23:01:59.711370 (+ 506us) write_op.cc:301] APPLY: starting
0620 23:01:59.711387 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:01:59.711420 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0620 23:01:59.711438 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:01:59.711615 (+ 177us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:01:59.711642 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:01:59.711646 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:01:59.711746 (+ 100us) log.cc:842] Serialized 22 byte log entry
0620 23:01:59.711861 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:01:59.711879 (+ 18us) write_op.cc:454] Released schema lock
0620 23:01:59.711885 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":896,"child_traces":[["op",{"apply.queue_time_us":432,"num_ops":1,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":1097,"prepare.run_wall_time_us":994150,"replication_time_us":605873,"thread_start_us":574,"threads_started":2,"wal-append.queue_time_us":433}]]}
I20260620 23:01:59.996336 14626 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:01:59.996588 14626 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:01:59.997578 14626 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:01:59.998250 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:01:59.998221 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:01:59.999049 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:00.624230 14627 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:00.624426 14627 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:00.625180 14627 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:00.625803 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:00.625851 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:00.626454 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:00.626770 14627 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:00.628029 14640 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=27, attempt_no=1}) took 2517 ms. Trace:
I20260620 23:02:00.628146 14640 rpcz_store.cc:276] 0620 23:01:58.110041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:58.110094 (+ 53us) service_pool.cc:227] Handling call
0620 23:02:00.627961 (+2517867us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:59.105502 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:59.105623 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:59.105630 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:01:59.105633 (+ 3us) tablet.cc:660] Decoding operations
0620 23:01:59.105727 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:59.105738 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:59.105742 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:59.105800 (+ 58us) tablet.cc:699] Row locks acquired
0620 23:01:59.105802 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:59.105850 (+ 48us) write_op.cc:270] Start()
0620 23:01:59.290005 (+184155us) write_op.cc:276] Timestamp: P: 1781996519105800 usec, L: 0
0620 23:01:59.290010 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:59.290239 (+ 229us) log.cc:842] Serialized 354 byte log entry
0620 23:02:00.626838 (+1336599us) op_driver.cc:444] REPLICATION: finished
0620 23:02:00.627269 (+ 431us) write_op.cc:301] APPLY: starting
0620 23:02:00.627284 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:00.627317 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:00.627342 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:00.627470 (+ 128us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:00.627501 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:00.627507 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:00.627625 (+ 118us) log.cc:842] Serialized 43 byte log entry
0620 23:02:00.627697 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:00.627714 (+ 17us) write_op.cc:454] Released schema lock
0620 23:02:00.627721 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":352000,"child_traces":[["op",{"apply.queue_time_us":378,"mutex_wait_us":61,"num_ops":1,"prepare.queue_time_us":994253,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":184813,"replication_time_us":1336801,"thread_start_us":243,"threads_started":1}]]}
I20260620 23:02:00.862986 14627 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:00.863188 14627 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:00.863943 14627 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:00.864363 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:00.864565 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:00.864892 14640 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=29, attempt_no=0}) took 1150 ms. Trace:
I20260620 23:02:00.865197 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:00.865036 14640 rpcz_store.cc:276] 0620 23:01:59.714130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:59.714196 (+ 66us) service_pool.cc:227] Handling call
0620 23:02:00.864824 (+1150628us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:01:59.714696 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:01:59.714788 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:01:59.714791 (+ 3us) write_op.cc:435] Acquired schema lock
0620 23:01:59.714793 (+ 2us) tablet.cc:660] Decoding operations
0620 23:01:59.714867 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:01:59.714875 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:01:59.714878 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:01:59.714917 (+ 39us) tablet.cc:699] Row locks acquired
0620 23:01:59.714919 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:01:59.714961 (+ 42us) write_op.cc:270] Start()
0620 23:01:59.896124 (+181163us) write_op.cc:276] Timestamp: P: 1781996519714907 usec, L: 0
0620 23:01:59.896129 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:01:59.896337 (+ 208us) log.cc:842] Serialized 354 byte log entry
0620 23:02:00.864139 (+967802us) op_driver.cc:444] REPLICATION: finished
0620 23:02:00.864273 (+ 134us) write_op.cc:301] APPLY: starting
0620 23:02:00.864291 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:00.864347 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:00.864388 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:00.864465 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:00.864501 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:00.864507 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:00.864627 (+ 120us) log.cc:842] Serialized 43 byte log entry
0620 23:02:00.864690 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:00.864707 (+ 17us) write_op.cc:454] Released schema lock
0620 23:02:00.864713 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":181752,"replication_time_us":967985}]]}
I20260620 23:02:00.865463 14627 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:01.150043 14627 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:01.150242 14627 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:01.151031 14627 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:01.151477 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:01.151687 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:01.152743 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:01.153015 14627 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:01.510926 14626 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:01.511404 14246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.131:51903 (request call id 603) took 1512 ms. Trace:
I20260620 23:02:01.511533 14246 rpcz_store.cc:276] 0620 23:01:59.998842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:01:59.998900 (+ 58us) service_pool.cc:227] Handling call
0620 23:01:59.999127 (+ 227us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:01:59.999327 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 30
0620 23:01:59.999625 (+ 298us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:01:59.999733 (+ 108us) write_op.cc:270] Start()
0620 23:02:00.669901 (+670168us) write_op.cc:276] Timestamp: P: 1781996519105800 usec, L: 0
0620 23:02:00.670601 (+ 700us) write_op.cc:270] Start()
0620 23:02:01.509764 (+839163us) write_op.cc:276] Timestamp: P: 1781996519714907 usec, L: 0
0620 23:02:01.510403 (+ 639us) log.cc:842] Serialized 708 byte log entry
0620 23:02:01.510748 (+ 345us) raft_consensus.cc:1631] Marking committed up to 30
0620 23:02:01.510757 (+ 9us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:02:01.510799 (+ 42us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:02:01.511305 (+ 506us) raft_consensus.cc:1671] finished
0620 23:02:01.511308 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:02:01.511368 (+ 60us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:00.670509 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:00.670634 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:00.670639 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:00.670641 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:00.670720 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:00.670732 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:00.670735 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:00.670778 (+ 43us) tablet.cc:699] Row locks acquired
0620 23:02:00.670779 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:02:01.510289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:01.510418 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:01.510423 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:01.510425 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:01.510502 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:01.510536 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:01.510539 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:01.510589 (+ 50us) tablet.cc:699] Row locks acquired
0620 23:02:01.510590 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":264,"threads_started":1,"wal-append.queue_time_us":333,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":424,"prepare.run_cpu_time_us":337,"prepare.run_wall_time_us":337,"thread_start_us":282,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":27,"prepare.queue_time_us":382,"prepare.run_cpu_time_us":361,"prepare.run_wall_time_us":363,"thread_start_us":248,"threads_started":1}]]}
I20260620 23:02:01.742537 14627 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:01.742738 14627 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:01.743460 14627 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:01.743930 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:01.743929 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:01.744643 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:01.744957 14627 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:01.952414 14627 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:01.952629 14627 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:01.953441 14627 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:01.953999 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:01.954217 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:01.954871 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:01.955148 14627 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:02.601892 14626 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:02.601961 14246 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.131:51903 (request call id 604) took 1088 ms. Trace:
I20260620 23:02:02.602170 14626 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:02.602165 14246 rpcz_store.cc:276] 0620 23:02:01.513786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:01.513849 (+ 63us) service_pool.cc:227] Handling call
0620 23:02:01.514052 (+ 203us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:02:01.514152 (+ 100us) raft_consensus.cc:1520] Early marking committed up to index 32
0620 23:02:01.514766 (+ 614us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:02:01.514910 (+ 144us) write_op.cc:270] Start()
0620 23:02:02.458040 (+943130us) write_op.cc:276] Timestamp: P: 1781996520630863 usec, L: 0
0620 23:02:02.458531 (+ 491us) write_op.cc:270] Start()
0620 23:02:02.600685 (+142154us) write_op.cc:276] Timestamp: P: 1781996521043863 usec, L: 0
0620 23:02:02.601071 (+ 386us) log.cc:842] Serialized 708 byte log entry
0620 23:02:02.601393 (+ 322us) raft_consensus.cc:1631] Marking committed up to 33
0620 23:02:02.601787 (+ 394us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:02:02.601822 (+ 35us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:02:02.601833 (+ 11us) raft_consensus.cc:1671] finished
0620 23:02:02.601835 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:02:02.601892 (+ 57us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:02.458463 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:02.458561 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:02.458566 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:02.458567 (+ 1us) tablet.cc:660] Decoding operations
0620 23:02:02.458640 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:02.458652 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:02.458654 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:02.458697 (+ 43us) tablet.cc:699] Row locks acquired
0620 23:02:02.458699 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:02.601427 (+142728us) op_driver.cc:444] REPLICATION: finished
0620 23:02:02.601837 (+ 410us) write_op.cc:301] APPLY: starting
0620 23:02:02.601851 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:02.601894 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:02.601896 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:02.602018 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:02.602139 (+ 121us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:02.602211 (+ 72us) write_op.cc:312] APPLY: finished
0620 23:02:02.602305 (+ 94us) log.cc:842] Serialized 43 byte log entry
Related trace 'op':
0620 23:02:02.600896 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:02.601057 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:02.601062 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:02.601064 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:02.601165 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:02.601177 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:02.601180 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:02.601264 (+ 84us) tablet.cc:699] Row locks acquired
0620 23:02:02.601269 (+ 5us) write_op.cc:260] PREPARE: finished
Metrics: {"spinlock_wait_cycles":46336,"thread_start_us":251,"threads_started":1,"wal-append.queue_time_us":368,"child_traces":[["op",{"apply.queue_time_us":360,"apply.run_cpu_time_us":663,"apply.run_wall_time_us":728,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":283,"prepare.run_cpu_time_us":330,"prepare.run_wall_time_us":329,"replication_time_us":1086505,"thread_start_us":457,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":541,"prepare.run_wall_time_us":542}]]}
I20260620 23:02:02.603017 14626 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:02.603461 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:02.603708 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:02.604297 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:03.321622 14626 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:03.324047 14660 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=33, attempt_no=0}) took 1576 ms. Trace:
I20260620 23:02:03.324186 14660 rpcz_store.cc:276] 0620 23:02:01.747281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:01.747350 (+ 69us) service_pool.cc:227] Handling call
0620 23:02:03.323991 (+1576641us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:01.857426 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:01.857553 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:01.857560 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:02:01.857564 (+ 4us) tablet.cc:660] Decoding operations
0620 23:02:01.857660 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:01.857671 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:01.857676 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:01.857739 (+ 63us) tablet.cc:699] Row locks acquired
0620 23:02:01.857741 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:01.857805 (+ 64us) write_op.cc:270] Start()
0620 23:02:02.450006 (+592201us) write_op.cc:276] Timestamp: P: 1781996521857740 usec, L: 0
0620 23:02:02.450011 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:02:02.450254 (+ 243us) log.cc:842] Serialized 354 byte log entry
0620 23:02:03.322999 (+872745us) op_driver.cc:444] REPLICATION: finished
0620 23:02:03.323412 (+ 413us) write_op.cc:301] APPLY: starting
0620 23:02:03.323424 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:03.323452 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:03.323502 (+ 50us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:03.323628 (+ 126us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:03.323655 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:03.323661 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:03.323760 (+ 99us) log.cc:842] Serialized 43 byte log entry
0620 23:02:03.323825 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:03.323839 (+ 14us) write_op.cc:454] Released schema lock
0620 23:02:03.323845 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":365,"num_ops":1,"prepare.queue_time_us":109605,"prepare.run_cpu_time_us":839,"prepare.run_wall_time_us":592925,"replication_time_us":872968,"thread_start_us":264,"threads_started":1}]]}
I20260620 23:02:03.438150 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:03.438371 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:03.439399 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:03.439875 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:03.439999 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:03.440721 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:03.694068 14626 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:03.694281 14626 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:03.695075 14626 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:03.695628 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:03.695911 14660 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=34, attempt_no=0}) took 1740 ms. Trace:
I20260620 23:02:03.696275 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:03.696040 14660 rpcz_store.cc:276] 0620 23:02:01.955917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:01.955969 (+ 52us) service_pool.cc:227] Handling call
0620 23:02:03.695842 (+1739873us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:02.450397 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:02.450526 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:02.450533 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:02:02.450535 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:02.450626 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:02.450640 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:02.450646 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:02.450704 (+ 58us) tablet.cc:699] Row locks acquired
0620 23:02:02.450706 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:02.450755 (+ 49us) write_op.cc:270] Start()
0620 23:02:02.940923 (+490168us) write_op.cc:276] Timestamp: P: 1781996522450668 usec, L: 0
0620 23:02:02.940928 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:02:02.941186 (+ 258us) log.cc:842] Serialized 354 byte log entry
0620 23:02:03.695138 (+753952us) op_driver.cc:444] REPLICATION: finished
0620 23:02:03.695258 (+ 120us) write_op.cc:301] APPLY: starting
0620 23:02:03.695277 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:03.695320 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:03.695358 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:03.695467 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:03.695503 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:03.695510 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:02:03.695626 (+ 116us) log.cc:842] Serialized 43 byte log entry
0620 23:02:03.695698 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:03.695715 (+ 17us) write_op.cc:454] Released schema lock
0620 23:02:03.695722 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"num_ops":1,"prepare.queue_time_us":494055,"prepare.run_cpu_time_us":816,"prepare.run_wall_time_us":490889,"replication_time_us":754180}]]}
I20260620 23:02:03.697075 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:03.697335 14626 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:03.883615 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:04.925580 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:04.925801 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:04.926501 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:04.927024 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:04.927413 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:04.928097 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:04.928741 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:04.929819 14670 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=35, attempt_no=0}) took 1603 ms. Trace:
I20260620 23:02:04.929940 14670 rpcz_store.cc:276] 0620 23:02:03.326464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:03.326541 (+ 77us) service_pool.cc:227] Handling call
0620 23:02:04.929776 (+1603235us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:03.327108 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:03.327202 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:03.327206 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:02:03.327207 (+ 1us) tablet.cc:660] Decoding operations
0620 23:02:03.327272 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:03.327279 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:03.327282 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:03.327320 (+ 38us) tablet.cc:699] Row locks acquired
0620 23:02:03.327321 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:02:03.327350 (+ 29us) write_op.cc:270] Start()
0620 23:02:04.243436 (+916086us) write_op.cc:276] Timestamp: P: 1781996523327296 usec, L: 0
0620 23:02:04.243440 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:02:04.243704 (+ 264us) log.cc:842] Serialized 354 byte log entry
0620 23:02:04.927410 (+683706us) op_driver.cc:444] REPLICATION: finished
0620 23:02:04.929216 (+ 1806us) write_op.cc:301] APPLY: starting
0620 23:02:04.929232 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:04.929268 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:04.929323 (+ 55us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:04.929420 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:04.929448 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:04.929453 (+ 5us) write_op.cc:312] APPLY: finished
0620 23:02:04.929561 (+ 108us) log.cc:842] Serialized 43 byte log entry
0620 23:02:04.929627 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:04.929643 (+ 16us) write_op.cc:454] Released schema lock
0620 23:02:04.929650 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1733,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":704,"prepare.run_wall_time_us":916721,"raft.queue_time_us":55,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":235,"replication_time_us":683940,"spinlock_wait_cycles":1792,"thread_start_us":1617,"threads_started":1}]]}
I20260620 23:02:05.079237 14665 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:05.079540 14665 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:05.080371 14665 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:05.080955 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:05.080989 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:05.081708 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:05.160776 14665 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:05.162494 14670 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=36, attempt_no=0}) took 1464 ms. Trace:
I20260620 23:02:05.162636 14670 rpcz_store.cc:276] 0620 23:02:03.698063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:03.698117 (+ 54us) service_pool.cc:227] Handling call
0620 23:02:05.162430 (+1464313us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:04.243869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:04.244009 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:04.244015 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:02:04.244017 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:04.244117 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:04.244129 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:04.244133 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:04.244189 (+ 56us) tablet.cc:699] Row locks acquired
0620 23:02:04.244192 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:02:04.244238 (+ 46us) write_op.cc:270] Start()
0620 23:02:04.935361 (+691123us) write_op.cc:276] Timestamp: P: 1781996524244193 usec, L: 0
0620 23:02:04.935365 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:02:04.935578 (+ 213us) log.cc:842] Serialized 354 byte log entry
0620 23:02:05.161724 (+226146us) op_driver.cc:444] REPLICATION: finished
0620 23:02:05.161848 (+ 124us) write_op.cc:301] APPLY: starting
0620 23:02:05.161872 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:05.161906 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:05.161929 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:05.162070 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:05.162105 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:05.162111 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:05.162217 (+ 106us) log.cc:842] Serialized 43 byte log entry
0620 23:02:05.162287 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:05.162309 (+ 22us) write_op.cc:454] Released schema lock
0620 23:02:05.162315 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57,"num_ops":1,"prepare.queue_time_us":545397,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":691773,"replication_time_us":226336}]]}
I20260620 23:02:05.792395 14675 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:05.792670 14675 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:05.793458 14675 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:05.794027 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:05.794032 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:05.794737 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:05.795421 14675 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:05.819679 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:05.819860 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:05.820665 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:05.821041 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:05.821810 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:05.822352 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:06.621775 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:06.672252 14675 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:06.672520 14675 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:06.673265 14675 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:06.673749 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:06.673872 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:06.674503 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:06.675112 14675 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:06.675513 14687 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=38, attempt_no=0}) took 1511 ms. Trace:
I20260620 23:02:06.675705 14687 rpcz_store.cc:276] 0620 23:02:05.164481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:05.164558 (+ 77us) service_pool.cc:227] Handling call
0620 23:02:06.675464 (+1510906us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:05.758416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:05.758554 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:05.758561 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:02:05.758563 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:05.758696 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:05.758707 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:05.758712 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:05.758767 (+ 55us) tablet.cc:699] Row locks acquired
0620 23:02:05.758769 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:05.758807 (+ 38us) write_op.cc:270] Start()
0620 23:02:05.848946 (+ 90139us) write_op.cc:276] Timestamp: P: 1781996525758759 usec, L: 0
0620 23:02:05.848953 (+ 7us) op_driver.cc:331] REPLICATION: starting
0620 23:02:05.849224 (+ 271us) log.cc:842] Serialized 354 byte log entry
0620 23:02:06.674400 (+825176us) op_driver.cc:444] REPLICATION: finished
0620 23:02:06.674869 (+ 469us) write_op.cc:301] APPLY: starting
0620 23:02:06.674887 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:06.674929 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:06.674957 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:06.675073 (+ 116us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:06.675105 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:06.675111 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:06.675213 (+ 102us) log.cc:842] Serialized 43 byte log entry
0620 23:02:06.675278 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:06.675296 (+ 18us) write_op.cc:454] Released schema lock
0620 23:02:06.675303 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":407,"num_ops":1,"prepare.queue_time_us":593409,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":90917,"raft.queue_time_us":39,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"replication_time_us":825421,"thread_start_us":298,"threads_started":1}]]}
I20260620 23:02:06.961599 14675 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:06.961838 14675 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:06.962692 14675 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:06.963172 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:06.963272 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:06.963842 14687 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=39, attempt_no=0}) took 1166 ms. Trace:
I20260620 23:02:06.963923 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:06.964217 14675 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:06.963971 14687 rpcz_store.cc:276] 0620 23:02:05.796852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:05.796920 (+ 68us) service_pool.cc:227] Handling call
0620 23:02:06.963787 (+1166867us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:05.849384 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:05.849525 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:05.849530 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:05.849532 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:05.849630 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:05.849641 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:05.849645 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:05.849702 (+ 57us) tablet.cc:699] Row locks acquired
0620 23:02:05.849705 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:02:05.849750 (+ 45us) write_op.cc:270] Start()
0620 23:02:06.359909 (+510159us) write_op.cc:276] Timestamp: P: 1781996525849689 usec, L: 0
0620 23:02:06.359914 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:02:06.360159 (+ 245us) log.cc:842] Serialized 354 byte log entry
0620 23:02:06.962621 (+602462us) op_driver.cc:444] REPLICATION: finished
0620 23:02:06.962743 (+ 122us) write_op.cc:301] APPLY: starting
0620 23:02:06.962762 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:06.962809 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:06.962846 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:06.962931 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:06.962969 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:06.962976 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:02:06.963063 (+ 87us) log.cc:842] Serialized 22 byte log entry
0620 23:02:06.963665 (+ 602us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:06.963684 (+ 19us) write_op.cc:454] Released schema lock
0620 23:02:06.963691 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"num_ops":1,"prepare.queue_time_us":52002,"prepare.run_cpu_time_us":806,"prepare.run_wall_time_us":510876,"replication_time_us":602635}]]}
I20260620 23:02:07.657995 14675 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:07.658221 14675 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:07.659017 14675 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:07.659515 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:07.659828 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:07.660374 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:07.660686 14675 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:08.219179 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:08.219408 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:08.219748 14116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.131:42907 (request call id 2026) took 1596 ms. Trace:
I20260620 23:02:08.220192 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:08.219894 14116 rpcz_store.cc:276] 0620 23:02:06.623734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:06.623799 (+ 65us) service_pool.cc:227] Handling call
0620 23:02:06.623971 (+ 172us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:02:06.624085 (+ 114us) raft_consensus.cc:1520] Early marking committed up to index 0
0620 23:02:06.624088 (+ 3us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:02:06.624217 (+ 129us) write_op.cc:270] Start()
0620 23:02:07.423403 (+799186us) write_op.cc:276] Timestamp: P: 1781996525758759 usec, L: 0
0620 23:02:07.423948 (+ 545us) write_op.cc:270] Start()
0620 23:02:08.218054 (+794106us) write_op.cc:276] Timestamp: P: 1781996525849689 usec, L: 0
0620 23:02:08.218668 (+ 614us) log.cc:842] Serialized 708 byte log entry
0620 23:02:08.219043 (+ 375us) raft_consensus.cc:1631] Marking committed up to 40
0620 23:02:08.219058 (+ 15us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:02:08.219098 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:02:08.219591 (+ 493us) raft_consensus.cc:1671] finished
0620 23:02:08.219593 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:02:08.219672 (+ 79us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:07.423866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:07.423972 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:07.423977 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:07.423978 (+ 1us) tablet.cc:660] Decoding operations
0620 23:02:07.424053 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:07.424067 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:07.424070 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:07.424114 (+ 44us) tablet.cc:699] Row locks acquired
0620 23:02:07.424115 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0620 23:02:08.218572 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:08.218686 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:08.218691 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:08.218693 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:08.218778 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:08.218790 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:08.218793 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:08.218841 (+ 48us) tablet.cc:699] Row locks acquired
0620 23:02:08.218842 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":268,"threads_started":1,"wal-append.queue_time_us":407,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":330,"prepare.run_cpu_time_us":376,"prepare.run_wall_time_us":376,"thread_start_us":216,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":388,"prepare.run_cpu_time_us":386,"prepare.run_wall_time_us":385,"thread_start_us":266,"threads_started":1}]]}
I20260620 23:02:08.220779 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:08.220821 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:08.221500 14051 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:08.221804 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:09.057510 14675 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:09.057708 14675 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:09.058411 14675 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:09.059070 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:09.059145 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:09.059787 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:09.060072 14675 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:09.060609 14703 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=41, attempt_no=0}) took 2094 ms. Trace:
I20260620 23:02:09.060709 14703 rpcz_store.cc:276] 0620 23:02:06.965729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:06.965805 (+ 76us) service_pool.cc:227] Handling call
0620 23:02:09.060567 (+2094762us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:07.156055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:07.156167 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:07.156171 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:02:07.156172 (+ 1us) tablet.cc:660] Decoding operations
0620 23:02:07.156244 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:07.156279 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:07.156283 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:07.156336 (+ 53us) tablet.cc:699] Row locks acquired
0620 23:02:07.156339 (+ 3us) write_op.cc:260] PREPARE: finished
0620 23:02:07.156388 (+ 49us) write_op.cc:270] Start()
0620 23:02:08.073550 (+917162us) write_op.cc:276] Timestamp: P: 1781996527156354 usec, L: 0
0620 23:02:08.073554 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:02:08.073839 (+ 285us) log.cc:842] Serialized 354 byte log entry
0620 23:02:09.059363 (+985524us) op_driver.cc:444] REPLICATION: finished
0620 23:02:09.059989 (+ 626us) write_op.cc:301] APPLY: starting
0620 23:02:09.060006 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:09.060046 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:09.060097 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:09.060187 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:09.060243 (+ 56us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:09.060249 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:09.060388 (+ 139us) log.cc:842] Serialized 43 byte log entry
0620 23:02:09.060442 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:09.060456 (+ 14us) write_op.cc:454] Released schema lock
0620 23:02:09.060461 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":564,"num_ops":1,"prepare.queue_time_us":189763,"prepare.run_cpu_time_us":811,"prepare.run_wall_time_us":917867,"replication_time_us":985784,"thread_start_us":454,"threads_started":1}]]}
I20260620 23:02:09.407478 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:09.407676 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:09.408123 14116 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.131:42907 (request call id 2027) took 1186 ms. Trace:
I20260620 23:02:09.408540 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:09.408926 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:09.408285 14116 rpcz_store.cc:276] 0620 23:02:08.222008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:08.222054 (+ 46us) service_pool.cc:227] Handling call
0620 23:02:08.222230 (+ 176us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:02:08.222329 (+ 99us) raft_consensus.cc:1520] Early marking committed up to index 42
0620 23:02:08.223008 (+ 679us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:02:08.223150 (+ 142us) write_op.cc:270] Start()
0620 23:02:09.208330 (+985180us) write_op.cc:276] Timestamp: P: 1781996526678587 usec, L: 0
0620 23:02:09.208895 (+ 565us) write_op.cc:270] Start()
0620 23:02:09.406128 (+197233us) write_op.cc:276] Timestamp: P: 1781996527156354 usec, L: 0
0620 23:02:09.406468 (+ 340us) log.cc:842] Serialized 708 byte log entry
0620 23:02:09.406944 (+ 476us) raft_consensus.cc:1631] Marking committed up to 43
0620 23:02:09.407295 (+ 351us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:02:09.407340 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:02:09.408027 (+ 687us) raft_consensus.cc:1671] finished
0620 23:02:09.408029 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:02:09.408072 (+ 43us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:09.208805 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:09.208957 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:09.208963 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:02:09.208964 (+ 1us) tablet.cc:660] Decoding operations
0620 23:02:09.209044 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:09.209058 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:09.209061 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:09.209114 (+ 53us) tablet.cc:699] Row locks acquired
0620 23:02:09.209116 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:09.406987 (+197871us) op_driver.cc:444] REPLICATION: finished
0620 23:02:09.407418 (+ 431us) write_op.cc:301] APPLY: starting
0620 23:02:09.407439 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:09.407488 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:09.407492 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:09.407577 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:09.407615 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:09.407648 (+ 33us) write_op.cc:312] APPLY: finished
0620 23:02:09.407743 (+ 95us) log.cc:842] Serialized 43 byte log entry
0620 23:02:09.407809 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:09.407824 (+ 15us) write_op.cc:454] Released schema lock
0620 23:02:09.407830 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0620 23:02:09.407919 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:09.407923 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0620 23:02:09.406342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:09.406519 (+ 177us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:09.406524 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:09.406526 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:09.406626 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:09.406639 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:09.406643 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:09.406693 (+ 50us) tablet.cc:699] Row locks acquired
0620 23:02:09.406694 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":392,"threads_started":1,"wal-append.queue_time_us":475,"child_traces":[["op",{"apply.queue_time_us":369,"apply.run_cpu_time_us":567,"apply.run_wall_time_us":567,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":341,"prepare.run_cpu_time_us":421,"prepare.run_wall_time_us":422,"replication_time_us":1183821,"thread_start_us":459,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":503,"prepare.run_wall_time_us":503}]]}
I20260620 23:02:09.409371 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:09.410045 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:09.410346 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:09.502009 14675 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:09.502249 14675 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:09.503042 14675 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:09.503685 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:09.503750 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:09.504309 14703 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=42, attempt_no=0}) took 1841 ms. Trace:
I20260620 23:02:09.504521 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:09.504434 14703 rpcz_store.cc:276] 0620 23:02:07.662693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:07.662762 (+ 69us) service_pool.cc:227] Handling call
0620 23:02:09.504255 (+1841493us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:08.073983 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:08.074107 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:08.074113 (+ 6us) write_op.cc:435] Acquired schema lock
0620 23:02:08.074115 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:08.074212 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:08.074225 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:08.074231 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:08.074289 (+ 58us) tablet.cc:699] Row locks acquired
0620 23:02:08.074291 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:08.074336 (+ 45us) write_op.cc:270] Start()
0620 23:02:08.440506 (+366170us) write_op.cc:276] Timestamp: P: 1781996528074272 usec, L: 0
0620 23:02:08.440511 (+ 5us) op_driver.cc:331] REPLICATION: starting
0620 23:02:08.440779 (+ 268us) log.cc:842] Serialized 354 byte log entry
0620 23:02:09.503443 (+1062664us) op_driver.cc:444] REPLICATION: finished
0620 23:02:09.503564 (+ 121us) write_op.cc:301] APPLY: starting
0620 23:02:09.503582 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:09.503629 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:09.503657 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:09.503766 (+ 109us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:09.503798 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:09.503804 (+ 6us) write_op.cc:312] APPLY: finished
0620 23:02:09.503915 (+ 111us) log.cc:842] Serialized 43 byte log entry
0620 23:02:09.503980 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:09.504004 (+ 24us) write_op.cc:454] Released schema lock
0620 23:02:09.504013 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"num_ops":1,"prepare.queue_time_us":410741,"prepare.run_cpu_time_us":839,"prepare.run_wall_time_us":366906,"replication_time_us":1062888}]]}
I20260620 23:02:09.504879 14675 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:09.694835 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:09.695024 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:09.695845 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:09.696167 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:09.696444 14246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:09.696926 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:09.697165 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:10.102833 14710 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:10.103061 14710 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:10.104065 14710 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827)
I20260620 23:02:10.104722 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:10.105106 14115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "0e509bac0552423badcb792c717c644b" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "d1788e51ad95470ab1cbd2588102d246" is_pre_election: true
I20260620 23:02:10.105608 14712 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36518 (ReqId={client: 03ad66bc2adc4b3c955a21bcfa128180, seq_no=43, attempt_no=0}) took 1042 ms. Trace:
I20260620 23:02:10.106150 14181 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e509bac0552423badcb792c717c644b; no voters: d1788e51ad95470ab1cbd2588102d246, fbe83de2708745dda707678c226dc34f
I20260620 23:02:10.105722 14712 rpcz_store.cc:276] 0620 23:02:09.062866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:09.062938 (+ 72us) service_pool.cc:227] Handling call
0620 23:02:10.105540 (+1042602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:09.063748 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 2644f1431b0848d98538f6b63c17382e
0620 23:02:09.063863 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:09.063866 (+ 3us) write_op.cc:435] Acquired schema lock
0620 23:02:09.063868 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:09.063959 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:09.063967 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:09.063970 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:09.504189 (+440219us) lock_manager.cc:599] Waited 440082us for lock on <redacted> (key hash 16040139233676677207)
0620 23:02:09.504201 (+ 12us) tablet.cc:699] Row locks acquired
0620 23:02:09.504203 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:09.504252 (+ 49us) write_op.cc:270] Start()
0620 23:02:10.018398 (+514146us) write_op.cc:276] Timestamp: P: 1781996529504192 usec, L: 0
0620 23:02:10.018402 (+ 4us) op_driver.cc:331] REPLICATION: starting
0620 23:02:10.018728 (+ 326us) log.cc:842] Serialized 354 byte log entry
0620 23:02:10.104001 (+ 85273us) op_driver.cc:444] REPLICATION: finished
0620 23:02:10.104424 (+ 423us) write_op.cc:301] APPLY: starting
0620 23:02:10.104470 (+ 46us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:10.104511 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:10.104514 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:10.105148 (+ 634us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:10.105187 (+ 39us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:10.105195 (+ 8us) write_op.cc:312] APPLY: finished
0620 23:02:10.105304 (+ 109us) log.cc:842] Serialized 43 byte log entry
0620 23:02:10.105370 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:10.105393 (+ 23us) write_op.cc:454] Released schema lock
0620 23:02:10.105401 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":354,"num_ops":1,"prepare.queue_time_us":433,"prepare.run_cpu_time_us":964,"prepare.run_wall_time_us":955087,"raft.queue_time_us":26,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":154,"replication_time_us":85570,"row_lock_wait_count":1,"row_lock_wait_us":440082,"spinlock_wait_cycles":350080,"thread_start_us":529,"threads_started":2}]]}
I20260620 23:02:10.106578 14710 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P 0e509bac0552423badcb792c717c644b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:10.203673 14226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:10.209409 14356 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260620 23:02:10.708913 14652 raft_consensus.cc:482] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fbe83de2708745dda707678c226dc34f)
I20260620 23:02:10.709112 14652 raft_consensus.cc:504] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe83de2708745dda707678c226dc34f" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 37463 } } peers { permanent_uuid: "d1788e51ad95470ab1cbd2588102d246" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 32827 } } peers { permanent_uuid: "0e509bac0552423badcb792c717c644b" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39957 } }
I20260620 23:02:10.709897 14652 leader_election.cc:288] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fbe83de2708745dda707678c226dc34f (127.10.134.131:37463), 0e509bac0552423badcb792c717c644b (127.10.134.130:39957)
I20260620 23:02:10.710302 14375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "fbe83de2708745dda707678c226dc34f" is_pre_election: true
I20260620 23:02:10.710544 14245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644f1431b0848d98538f6b63c17382e" candidate_uuid: "d1788e51ad95470ab1cbd2588102d246" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "0e509bac0552423badcb792c717c644b" is_pre_election: true
I20260620 23:02:10.711555 14050 leader_election.cc:302] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1788e51ad95470ab1cbd2588102d246; no voters: 0e509bac0552423badcb792c717c644b, fbe83de2708745dda707678c226dc34f
I20260620 23:02:10.711874 14652 raft_consensus.cc:2726] T 2644f1431b0848d98538f6b63c17382e P d1788e51ad95470ab1cbd2588102d246 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:10.713040 14096 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e718ce3388774d5488cd92156d9d23be | 127.10.134.190:38903 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:41659 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
0e509bac0552423badcb792c717c644b | 127.10.134.130:39957 | HEALTHY | <none> | 0 | 0
d1788e51ad95470ab1cbd2588102d246 | 127.10.134.129:32827 | HEALTHY | <none> | 0 | 0
fbe83de2708745dda707678c226dc34f | 127.10.134.131:37463 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:32827
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:39957
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:37463
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:32827
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:39957
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:37463
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:41659 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260620 23:02:10.753279 10778 log_verifier.cc:126] Checking tablet 2644f1431b0848d98538f6b63c17382e
I20260620 23:02:10.816483 10778 log_verifier.cc:177] Verified matching terms for 46 ops in tablet 2644f1431b0848d98538f6b63c17382e
I20260620 23:02:10.882910 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 14034
I20260620 23:02:10.892413 14156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:11.119855 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 14034
W20260620 23:02:11.149371 14311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.134.129:32827 (error 108) [suppressed 2 similar messages]
I20260620 23:02:11.149964 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 14164
W20260620 23:02:11.151237 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:11.158407 14286 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 23:02:11.167290 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260620 23:02:11.178259 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260620 23:02:11.189591 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260620 23:02:11.201365 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260620 23:02:11.212960 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260620 23:02:11.224258 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260620 23:02:11.235751 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260620 23:02:11.247256 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260620 23:02:11.258626 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260620 23:02:11.270197 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260620 23:02:11.281813 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260620 23:02:11.293264 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260620 23:02:11.304733 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260620 23:02:11.316377 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260620 23:02:11.327963 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260620 23:02:11.339057 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 14164
W20260620 23:02:11.342967 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260620 23:02:11.354084 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20260620 23:02:11.365392 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20260620 23:02:11.369262 14310 connection.cc:535] client connection to 127.10.134.130:39957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:02:11.369812 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 14294
W20260620 23:02:11.370157 14310 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer 0e509bac0552423badcb792c717c644b (127.10.134.130:39957): Couldn't send request to peer 0e509bac0552423badcb792c717c644b. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.130:39957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:11.378525 14416 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 23:02:11.509481 14311 consensus_peers.cc:487] T 2644f1431b0848d98538f6b63c17382e P fbe83de2708745dda707678c226dc34f -> Peer d1788e51ad95470ab1cbd2588102d246 (127.10.134.129:32827): Couldn't send request to peer d1788e51ad95470ab1cbd2588102d246. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.129:32827: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20260620 23:02:11.509557 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 14294
I20260620 23:02:11.533844 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 13944
I20260620 23:02:11.534873 14005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:11.642222 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 13944
2026-06-20T23:02:11Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (34497 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20260620 23:02:11.681464 10778 test_util.cc:268] Using random seed: 1715669157
I20260620 23:02:11.683244 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:11.683305 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:11.683346 10778 ts_itest-base.cc:114] 5 tablet servers
I20260620 23:02:11.683383 10778 ts_itest-base.cc:115] 5 replicas per TS
I20260620 23:02:11.683429 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:11Z Disabled control of system clock
I20260620 23:02:11.709221 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:44137
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:44137 with env {}
W20260620 23:02:12.021078 14773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:12.021402 14773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:12.021555 14773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:12.028573 14773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:12.028800 14773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:12.028913 14773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:12.029014 14773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:12.038203 14773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:44137
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:12.039494 14773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:12.041209 14773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:12.052783 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:12.052948 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:12.052922 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:12.053843 14773 server_base.cc:1034] running on GCE node
I20260620 23:02:12.054545 14773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:12.055775 14773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:12.057004 14773 hybrid_clock.cc:648] HybridClock initialized: now 1781996532056929 us; error 49 us; skew 500 ppm
I20260620 23:02:12.057348 14773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:12.059558 14773 webserver.cc:460] Webserver started at http://127.10.134.190:40807/ using document root <none> and password file <none>
I20260620 23:02:12.060220 14773 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:12.060343 14773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:12.060631 14773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:12.063241 14773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0603d265b06b4184be4a1a58d2ce603b"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:12.063999 14773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0603d265b06b4184be4a1a58d2ce603b"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:12.068773 14773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 23:02:12.071933 14787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:12.073226 14773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:12.073403 14773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0603d265b06b4184be4a1a58d2ce603b"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:12.073593 14773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:12.093215 14773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:12.093981 14773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:12.094255 14773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:12.111493 14773 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:44137
I20260620 23:02:12.111515 14838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:44137 every 8 connection(s)
I20260620 23:02:12.112766 14773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:12.115111 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 14773
I20260620 23:02:12.115294 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:12.116901 14839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:12.129499 14839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b: Bootstrap starting.
I20260620 23:02:12.132369 14839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:12.133322 14839 log.cc:824] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:12.135784 14839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b: No bootstrap required, opened a new log
I20260620 23:02:12.142779 14839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0603d265b06b4184be4a1a58d2ce603b" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44137 } }
I20260620 23:02:12.143191 14839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:12.143290 14839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0603d265b06b4184be4a1a58d2ce603b, State: Initialized, Role: FOLLOWER
I20260620 23:02:12.143869 14839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0603d265b06b4184be4a1a58d2ce603b" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44137 } }
I20260620 23:02:12.144101 14839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:12.144246 14839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:12.144407 14839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:12.146468 14839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0603d265b06b4184be4a1a58d2ce603b" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44137 } }
I20260620 23:02:12.146925 14839 leader_election.cc:302] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0603d265b06b4184be4a1a58d2ce603b; no voters:
I20260620 23:02:12.147477 14839 leader_election.cc:288] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:12.147610 14844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:12.148259 14844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [term 1 LEADER]: Becoming Leader. State: Replica: 0603d265b06b4184be4a1a58d2ce603b, State: Running, Role: LEADER
I20260620 23:02:12.148756 14844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0603d265b06b4184be4a1a58d2ce603b" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44137 } }
I20260620 23:02:12.149389 14839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:12.151541 14846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0603d265b06b4184be4a1a58d2ce603b. Latest consensus state: current_term: 1 leader_uuid: "0603d265b06b4184be4a1a58d2ce603b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0603d265b06b4184be4a1a58d2ce603b" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44137 } } }
I20260620 23:02:12.151953 14846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:12.152017 14845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0603d265b06b4184be4a1a58d2ce603b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0603d265b06b4184be4a1a58d2ce603b" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44137 } } }
I20260620 23:02:12.152750 14845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:12.153545 14849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:12.157971 14849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:12.166175 14849 catalog_manager.cc:1347] Generated new cluster ID: be5f3d90684343c6a6f27ed4e5f7277d
I20260620 23:02:12.166292 14849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:12.174646 14849 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:12.175791 14849 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:12.183796 14849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0603d265b06b4184be4a1a58d2ce603b: Generated new TSK 0
I20260620 23:02:12.184523 14849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:12.194689 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260620 23:02:12.521838 14863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:12.522174 14863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:12.522320 14863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:12.529239 14863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:12.529436 14863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:12.540126 14863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:12.541718 14863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:12.544075 14863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:12.557451 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:12.557397 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:12.557480 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:12.558575 14863 server_base.cc:1034] running on GCE node
I20260620 23:02:12.559134 14863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:12.559965 14863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:12.561177 14863 hybrid_clock.cc:648] HybridClock initialized: now 1781996532561095 us; error 53 us; skew 500 ppm
I20260620 23:02:12.561457 14863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:12.563660 14863 webserver.cc:460] Webserver started at http://127.10.134.129:33281/ using document root <none> and password file <none>
I20260620 23:02:12.564330 14863 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:12.564458 14863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:12.564740 14863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:12.567224 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d18d528e165543558ff363a3dc7f7463"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:12.567947 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d18d528e165543558ff363a3dc7f7463"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:12.572675 14863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:02:12.575843 14877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:12.577230 14863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 23:02:12.577390 14863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d18d528e165543558ff363a3dc7f7463"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:12.577560 14863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:12.601228 14863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:12.602049 14863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:12.602353 14863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:12.603266 14863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:12.604611 14863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:12.604681 14863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:12.604847 14863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:12.604910 14863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:12.636818 14863 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:35377
I20260620 23:02:12.636863 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:35377 every 8 connection(s)
I20260620 23:02:12.638128 14863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:12.645673 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 14863
I20260620 23:02:12.645881 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:12.649224 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260620 23:02:12.652237 14990 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:12.652532 14990 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:12.653366 14990 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:12.655506 14804 ts_manager.cc:194] Registered new tserver with Master: d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377)
I20260620 23:02:12.657624 14804 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:45687
W20260620 23:02:12.959174 14994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:12.959434 14994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:12.959513 14994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:12.966385 14994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:12.966524 14994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:12.974924 14994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:12.976001 14994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:12.977710 14994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:12.990115 14999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:12.990219 15000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:12.990216 15002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:12.991127 14994 server_base.cc:1034] running on GCE node
I20260620 23:02:12.991679 14994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:12.992471 14994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:12.993674 14994 hybrid_clock.cc:648] HybridClock initialized: now 1781996532993600 us; error 50 us; skew 500 ppm
I20260620 23:02:12.993988 14994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:12.996181 14994 webserver.cc:460] Webserver started at http://127.10.134.130:44611/ using document root <none> and password file <none>
I20260620 23:02:12.996775 14994 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:12.996879 14994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:12.997118 14994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:12.999588 14994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "aaeacf1e4563412da1e4d6423583dc62"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:13.000272 14994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "aaeacf1e4563412da1e4d6423583dc62"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:13.005143 14994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:02:13.008450 15008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.009698 14994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:13.009883 14994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aaeacf1e4563412da1e4d6423583dc62"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:13.010120 14994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:13.029785 14994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:13.030592 14994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:13.030901 14994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:13.031764 14994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:13.033087 14994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:13.033183 14994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.033319 14994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:13.033387 14994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.065220 14994 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:37759
I20260620 23:02:13.065258 15120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:37759 every 8 connection(s)
I20260620 23:02:13.066504 14994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:13.069480 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 14994
I20260620 23:02:13.069624 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:13.073212 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260620 23:02:13.083645 15121 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:13.083956 15121 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:13.084669 15121 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:13.086012 14804 ts_manager.cc:194] Registered new tserver with Master: aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759)
I20260620 23:02:13.086783 14804 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:41549
W20260620 23:02:13.389664 15124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:13.389993 15124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:13.390103 15124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:13.397178 15124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:13.397336 15124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:13.406258 15124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:13.407421 15124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:13.409017 15124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:13.421164 15130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:13.421240 15129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:13.421243 15132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:13.422288 15124 server_base.cc:1034] running on GCE node
I20260620 23:02:13.422837 15124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:13.423691 15124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:13.424887 15124 hybrid_clock.cc:648] HybridClock initialized: now 1781996533424817 us; error 56 us; skew 500 ppm
I20260620 23:02:13.425120 15124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:13.427179 15124 webserver.cc:460] Webserver started at http://127.10.134.131:46635/ using document root <none> and password file <none>
I20260620 23:02:13.427783 15124 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:13.427882 15124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:13.428117 15124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:13.430511 15124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1798ec3af4194596838b8d95d1905524"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:13.431178 15124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1798ec3af4194596838b8d95d1905524"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:13.436017 15124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:02:13.439150 15138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.440412 15124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:13.440567 15124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1798ec3af4194596838b8d95d1905524"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:13.440742 15124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:13.457113 15124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:13.457906 15124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:13.458249 15124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:13.459172 15124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:13.460544 15124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:13.460644 15124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.460779 15124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:13.460857 15124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.492723 15124 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:41013
I20260620 23:02:13.492755 15250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:41013 every 8 connection(s)
I20260620 23:02:13.494078 15124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:13.494422 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 15124
I20260620 23:02:13.494623 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:13.498050 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.132:0
--local_ip_for_outbound_sockets=127.10.134.132
--webserver_interface=127.10.134.132
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260620 23:02:13.508359 15251 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:13.508666 15251 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:13.509346 15251 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:13.510730 14804 ts_manager.cc:194] Registered new tserver with Master: 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013)
I20260620 23:02:13.512036 14804 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:46739
I20260620 23:02:13.660495 14990 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
W20260620 23:02:13.818058 15254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:13.818313 15254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:13.818498 15254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:13.825700 15254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:13.825904 15254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.132
I20260620 23:02:13.834805 15254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.134.132
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.132
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:13.836148 15254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:13.837782 15254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:13.850785 15259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:13.850795 15260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:13.850996 15262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:13.852174 15254 server_base.cc:1034] running on GCE node
I20260620 23:02:13.852805 15254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:13.853703 15254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:13.854950 15254 hybrid_clock.cc:648] HybridClock initialized: now 1781996533854858 us; error 70 us; skew 500 ppm
I20260620 23:02:13.855237 15254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:13.857419 15254 webserver.cc:460] Webserver started at http://127.10.134.132:41299/ using document root <none> and password file <none>
I20260620 23:02:13.858136 15254 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:13.858268 15254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:13.858565 15254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:13.861048 15254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a6d18b29646f40ac885810025578f279"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:13.861744 15254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a6d18b29646f40ac885810025578f279"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:13.866535 15254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:02:13.869714 15268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.871061 15254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:13.871234 15254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a6d18b29646f40ac885810025578f279"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:13.871431 15254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:13.885289 15254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:13.886116 15254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:13.886404 15254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:13.887274 15254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:13.888618 15254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:13.888715 15254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.888856 15254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:13.888934 15254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:13.920811 15254 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.132:43969
I20260620 23:02:13.920871 15380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.132:43969 every 8 connection(s)
I20260620 23:02:13.922106 15254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260620 23:02:13.927325 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 15254
I20260620 23:02:13.927469 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260620 23:02:13.930423 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.133:0
--local_ip_for_outbound_sockets=127.10.134.133
--webserver_interface=127.10.134.133
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260620 23:02:13.936692 15381 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:13.936991 15381 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:13.937682 15381 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:13.939116 14804 ts_manager.cc:194] Registered new tserver with Master: a6d18b29646f40ac885810025578f279 (127.10.134.132:43969)
I20260620 23:02:13.939872 14804 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.132:51055
I20260620 23:02:14.089269 15121 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
W20260620 23:02:14.244917 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:14.245163 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:14.245240 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:14.252220 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:14.252365 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.133
I20260620 23:02:14.261055 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.134.133
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.133
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:14.262173 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:14.263731 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:14.275853 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:14.275882 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:14.276368 15392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:14.277024 15384 server_base.cc:1034] running on GCE node
I20260620 23:02:14.277673 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:14.278626 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:14.279846 15384 hybrid_clock.cc:648] HybridClock initialized: now 1781996534279781 us; error 46 us; skew 500 ppm
I20260620 23:02:14.280150 15384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:14.282466 15384 webserver.cc:460] Webserver started at http://127.10.134.133:38945/ using document root <none> and password file <none>
I20260620 23:02:14.283097 15384 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:14.283226 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:14.283512 15384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:14.286028 15384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b9b038b3d62a402a8acee576071f7fa6"
format_stamp: "Formatted at 2026-06-20 23:02:14 on dist-test-slave-8k25"
I20260620 23:02:14.286746 15384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b9b038b3d62a402a8acee576071f7fa6"
format_stamp: "Formatted at 2026-06-20 23:02:14 on dist-test-slave-8k25"
I20260620 23:02:14.291601 15384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:02:14.294942 15398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:14.296243 15384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:02:14.296412 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b9b038b3d62a402a8acee576071f7fa6"
format_stamp: "Formatted at 2026-06-20 23:02:14 on dist-test-slave-8k25"
I20260620 23:02:14.296602 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:14.309489 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:14.310264 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:14.310559 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:14.311434 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:14.312774 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:14.312872 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:14.313014 15384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:14.313089 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:14.345423 15384 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.133:38093
I20260620 23:02:14.345470 15510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.133:38093 every 8 connection(s)
I20260620 23:02:14.346740 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260620 23:02:14.350520 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 15384
I20260620 23:02:14.350697 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260620 23:02:14.360276 15511 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:14.360589 15511 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:14.361269 15511 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:14.362512 14804 ts_manager.cc:194] Registered new tserver with Master: b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:14.363245 14804 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.133:50977
I20260620 23:02:14.367106 10778 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260620 23:02:14.382663 14804 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32792:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260620 23:02:14.389142 14804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260620 23:02:14.428035 15186 tablet_service.cc:1467] Processing CreateTablet for tablet 006a43f02e7c4304a86b1a72aa1ab5e5 (DEFAULT_TABLE table=TestTable [id=20239c6c99e143b2ac7b4d41ca30ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:14.429365 15186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006a43f02e7c4304a86b1a72aa1ab5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:14.431145 15446 tablet_service.cc:1467] Processing CreateTablet for tablet 006a43f02e7c4304a86b1a72aa1ab5e5 (DEFAULT_TABLE table=TestTable [id=20239c6c99e143b2ac7b4d41ca30ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:14.432538 15446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006a43f02e7c4304a86b1a72aa1ab5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:14.434890 14925 tablet_service.cc:1467] Processing CreateTablet for tablet 006a43f02e7c4304a86b1a72aa1ab5e5 (DEFAULT_TABLE table=TestTable [id=20239c6c99e143b2ac7b4d41ca30ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:14.436269 14925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006a43f02e7c4304a86b1a72aa1ab5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:14.456521 15535 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: Bootstrap starting.
I20260620 23:02:14.456157 15316 tablet_service.cc:1467] Processing CreateTablet for tablet 006a43f02e7c4304a86b1a72aa1ab5e5 (DEFAULT_TABLE table=TestTable [id=20239c6c99e143b2ac7b4d41ca30ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:14.456869 15056 tablet_service.cc:1467] Processing CreateTablet for tablet 006a43f02e7c4304a86b1a72aa1ab5e5 (DEFAULT_TABLE table=TestTable [id=20239c6c99e143b2ac7b4d41ca30ca93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:14.457590 15316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006a43f02e7c4304a86b1a72aa1ab5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:14.458740 15056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 006a43f02e7c4304a86b1a72aa1ab5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:14.462234 15535 tablet_bootstrap.cc:654] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:14.463522 15535 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:14.466995 15533 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524: Bootstrap starting.
I20260620 23:02:14.468256 15534 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: Bootstrap starting.
I20260620 23:02:14.470513 15533 tablet_bootstrap.cc:654] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:14.471721 15533 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:14.472869 15535 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: No bootstrap required, opened a new log
I20260620 23:02:14.473158 15535 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20260620 23:02:14.478864 15534 tablet_bootstrap.cc:654] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:14.480134 15534 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:14.482664 15538 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Bootstrap starting.
I20260620 23:02:14.485646 15533 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524: No bootstrap required, opened a new log
I20260620 23:02:14.485952 15533 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20260620 23:02:14.486331 15538 tablet_bootstrap.cc:654] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:14.487576 15538 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:14.490298 15535 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.491117 15535 raft_consensus.cc:374] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:14.491245 15535 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18d528e165543558ff363a3dc7f7463, State: Initialized, Role: FOLLOWER
I20260620 23:02:14.491887 15535 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.492520 15534 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: No bootstrap required, opened a new log
I20260620 23:02:14.492839 15534 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20260620 23:02:14.496537 15538 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: No bootstrap required, opened a new log
I20260620 23:02:14.496799 15538 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260620 23:02:14.498692 15539 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Bootstrap starting.
I20260620 23:02:14.503504 15539 tablet_bootstrap.cc:654] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:14.503780 15535 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: Time spent starting tablet: real 0.031s user 0.016s sys 0.001s
I20260620 23:02:14.504745 15539 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:14.504508 15534 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.504933 15534 raft_consensus.cc:374] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:14.505060 15534 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b038b3d62a402a8acee576071f7fa6, State: Initialized, Role: FOLLOWER
I20260620 23:02:14.505728 15534 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.513759 15539 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: No bootstrap required, opened a new log
I20260620 23:02:14.514133 15539 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260620 23:02:14.514835 15251 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
I20260620 23:02:14.517022 15538 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.517524 15538 raft_consensus.cc:374] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:14.517664 15538 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d18b29646f40ac885810025578f279, State: Initialized, Role: FOLLOWER
I20260620 23:02:14.518313 15511 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
I20260620 23:02:14.518503 15538 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.518949 15534 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: Time spent starting tablet: real 0.026s user 0.011s sys 0.011s
I20260620 23:02:14.524209 15381 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
I20260620 23:02:14.524816 15538 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Time spent starting tablet: real 0.028s user 0.013s sys 0.004s
I20260620 23:02:14.524725 15533 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.525218 15533 raft_consensus.cc:374] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:14.525321 15533 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1798ec3af4194596838b8d95d1905524, State: Initialized, Role: FOLLOWER
I20260620 23:02:14.525981 15533 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.527426 15539 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.527815 15533 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524: Time spent starting tablet: real 0.042s user 0.006s sys 0.008s
I20260620 23:02:14.527973 15539 raft_consensus.cc:374] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:14.528121 15539 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaeacf1e4563412da1e4d6423583dc62, State: Initialized, Role: FOLLOWER
I20260620 23:02:14.528725 15539 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.530509 15539 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
W20260620 23:02:14.569175 15122 tablet.cc:2369] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:14.574604 15545 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:14.574882 15545 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.576831 15545 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:14.589963 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62" is_pre_election: true
I20260620 23:02:14.590560 15076 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 0.
I20260620 23:02:14.590638 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:14.591190 15206 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 0.
I20260620 23:02:14.591411 15466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:14.591920 15466 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 0.
I20260620 23:02:14.591894 15271 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, a6d18b29646f40ac885810025578f279, aaeacf1e4563412da1e4d6423583dc62; no voters:
I20260620 23:02:14.592329 15545 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:14.592501 15545 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:14.592597 15545 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:14.594813 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:14.595254 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 0.
I20260620 23:02:14.595219 15545 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.596011 15545 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [CANDIDATE]: Term 1 election: Requested vote from peers 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:14.596338 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62"
I20260620 23:02:14.596395 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524"
I20260620 23:02:14.596446 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463"
I20260620 23:02:14.596549 15076 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:14.596560 15206 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:14.596585 15466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "a6d18b29646f40ac885810025578f279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6"
I20260620 23:02:14.596622 14945 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:14.596762 15466 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:14.599606 15206 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 1.
I20260620 23:02:14.599643 15076 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 1.
I20260620 23:02:14.599689 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 1.
I20260620 23:02:14.599713 15466 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6d18b29646f40ac885810025578f279 in term 1.
W20260620 23:02:14.600230 15512 tablet.cc:2369] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:14.600309 15271 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: a6d18b29646f40ac885810025578f279, b9b038b3d62a402a8acee576071f7fa6, d18d528e165543558ff363a3dc7f7463; no voters:
I20260620 23:02:14.600646 15545 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:14.601243 15545 raft_consensus.cc:686] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 1 LEADER]: Becoming Leader. State: Replica: a6d18b29646f40ac885810025578f279, State: Running, Role: LEADER
I20260620 23:02:14.601701 15545 consensus_queue.cc:237] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:14.606604 14804 catalog_manager.cc:5495] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6d18b29646f40ac885810025578f279 (127.10.134.132). New cstate: current_term: 1 leader_uuid: "a6d18b29646f40ac885810025578f279" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:14.626470 10778 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260620 23:02:14.628401 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d18d528e165543558ff363a3dc7f7463 to finish bootstrapping
I20260620 23:02:14.637338 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver aaeacf1e4563412da1e4d6423583dc62 to finish bootstrapping
W20260620 23:02:14.641099 14991 tablet.cc:2369] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:14.645920 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1798ec3af4194596838b8d95d1905524 to finish bootstrapping
I20260620 23:02:14.653746 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a6d18b29646f40ac885810025578f279 to finish bootstrapping
I20260620 23:02:14.661886 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b9b038b3d62a402a8acee576071f7fa6 to finish bootstrapping
W20260620 23:02:14.674201 15382 tablet.cc:2369] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:14.679103 10778 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20260620 23:02:14.698011 15206 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Refusing update from remote peer a6d18b29646f40ac885810025578f279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:14.698011 14945 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 1 FOLLOWER]: Refusing update from remote peer a6d18b29646f40ac885810025578f279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:14.698014 15466 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Refusing update from remote peer a6d18b29646f40ac885810025578f279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:14.698254 15076 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Refusing update from remote peer a6d18b29646f40ac885810025578f279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:14.699080 15545 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:14.699378 15552 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:14.699702 15558 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:14.700099 15559 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:14.726342 15557 mvcc.cc:205] Tried to move back new op lower bound from 7299057806112411648 to 7299057805733736448. Current Snapshot: MvccSnapshot[applied={T|T < 7299057806112411648}]
I20260620 23:02:14.734935 15560 mvcc.cc:205] Tried to move back new op lower bound from 7299057806112411648 to 7299057805733736448. Current Snapshot: MvccSnapshot[applied={T|T < 7299057806112411648}]
W20260620 23:02:14.751590 15270 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.10.134.133:38093 timed out after 0.050s (SENT)
I20260620 23:02:14.765681 15465 raft_consensus.cc:1203] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 0.0->[1.1-1.2] Dedup: 1.2->[]
W20260620 23:02:14.766287 15466 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.10.134.132:43617 (request call id 3) took 65 ms (client timeout 50 ms). Trace:
W20260620 23:02:14.766429 15466 rpcz_store.cc:269] 0620 23:02:14.700926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:14.700974 (+ 48us) service_pool.cc:227] Handling call
0620 23:02:14.701429 (+ 455us) raft_consensus.cc:1457] Updating replica for 2 ops
0620 23:02:14.701804 (+ 375us) raft_consensus.cc:1520] Early marking committed up to index 0
0620 23:02:14.701816 (+ 12us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0620 23:02:14.702607 (+ 791us) write_op.cc:270] Start()
0620 23:02:14.702819 (+ 212us) write_op.cc:276] Timestamp: P: 1781996534695413 usec, L: 0
0620 23:02:14.729409 (+ 26590us) log.cc:842] Serialized 670 byte log entry
0620 23:02:14.763016 (+ 33607us) raft_consensus.cc:1631] Marking committed up to 0
0620 23:02:14.763037 (+ 21us) raft_consensus.cc:1683] Filling consensus response to leader.
0620 23:02:14.765497 (+ 2460us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0620 23:02:14.765523 (+ 26us) raft_consensus.cc:1671] finished
0620 23:02:14.765526 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0620 23:02:14.766204 (+ 678us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:14.725571 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 006a43f02e7c4304a86b1a72aa1ab5e5
0620 23:02:14.725780 (+ 209us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:14.725797 (+ 17us) write_op.cc:435] Acquired schema lock
0620 23:02:14.725800 (+ 3us) tablet.cc:660] Decoding operations
0620 23:02:14.726600 (+ 800us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:14.726672 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:14.726689 (+ 17us) tablet.cc:683] Acquiring locks for 10 operations
0620 23:02:14.727862 (+ 1173us) tablet.cc:699] Row locks acquired
0620 23:02:14.727867 (+ 5us) write_op.cc:260] PREPARE: finished
0620 23:02:14.767613 (+ 39746us) op_driver.cc:444] REPLICATION: finished
Metrics: {"raft.queue_time_us":2605,"raft.run_cpu_time_us":43,"raft.run_wall_time_us":44,"thread_start_us":33350,"threads_started":1,"wal-append.queue_time_us":33490,"child_traces":[["op",{"num_ops":10,"partition_lock_wait_us":45,"prepare.queue_time_us":21322,"prepare.run_cpu_time_us":2895,"prepare.run_wall_time_us":2894,"replication_time_us":64976,"thread_start_us":21192,"threads_started":2}]]}
I20260620 23:02:14.798025 10778 raft_consensus_election-itest.cc:408] Killing current leader a6d18b29646f40ac885810025578f279...
I20260620 23:02:14.798205 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 15254
I20260620 23:02:14.799463 15376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:14.891016 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 15254
I20260620 23:02:14.907572 10778 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260620 23:02:14.910145 10778 test_util.cc:268] Using random seed: 1718897836
I20260620 23:02:15.039209 10778 test_util.cc:268] Using random seed: 1719026875
I20260620 23:02:15.247161 10778 test_util.cc:268] Using random seed: 1719234844
I20260620 23:02:15.555107 10778 test_util.cc:268] Using random seed: 1719542767
I20260620 23:02:15.962761 10778 test_util.cc:268] Using random seed: 1719950448
I20260620 23:02:16.298820 15577 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a6d18b29646f40ac885810025578f279)
I20260620 23:02:16.298758 15576 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a6d18b29646f40ac885810025578f279)
I20260620 23:02:16.299019 15577 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:16.299049 15576 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:16.300812 15577 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377)
I20260620 23:02:16.301123 15578 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a6d18b29646f40ac885810025578f279)
I20260620 23:02:16.301431 15578 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
W20260620 23:02:16.317154 15009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.329739 15578 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:16.330217 15576 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
W20260620 23:02:16.348120 15139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
W20260620 23:02:16.355414 15009 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.364333 15579 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a6d18b29646f40ac885810025578f279)
I20260620 23:02:16.364605 15579 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
W20260620 23:02:16.375519 15399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
W20260620 23:02:16.380582 15139 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.381438 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:16.381778 15465 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1798ec3af4194596838b8d95d1905524 in term 1.
I20260620 23:02:16.382004 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:16.382257 15206 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 1.
I20260620 23:02:16.382664 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:16.382946 15205 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 1.
W20260620 23:02:16.385892 15399 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.386565 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62" is_pre_election: true
I20260620 23:02:16.386860 15075 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 1.
I20260620 23:02:16.387418 15399 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6; no voters: a6d18b29646f40ac885810025578f279
I20260620 23:02:16.388025 15577 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:02:16.388141 15577 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a6d18b29646f40ac885810025578f279)
I20260620 23:02:16.388221 15577 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:16.389396 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62" is_pre_election: true
I20260620 23:02:16.389592 15075 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1798ec3af4194596838b8d95d1905524 in term 1.
I20260620 23:02:16.390080 15139 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6; no voters: a6d18b29646f40ac885810025578f279
I20260620 23:02:16.390442 15578 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:02:16.390563 15578 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a6d18b29646f40ac885810025578f279)
I20260620 23:02:16.390643 15578 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:16.391342 15577 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:16.392150 15577 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 2 election: Requested vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377)
I20260620 23:02:16.393537 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62"
I20260620 23:02:16.393452 15578 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:16.393718 15075 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:16.394783 15578 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 2 election: Requested vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:16.395735 15076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62"
I20260620 23:02:16.396458 15075 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 2.
W20260620 23:02:16.397565 15139 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.397543 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524"
I20260620 23:02:16.397837 15205 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b038b3d62a402a8acee576071f7fa6 in current term 2: Already voted for candidate 1798ec3af4194596838b8d95d1905524 in this term.
I20260620 23:02:16.398138 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6"
I20260620 23:02:16.398423 15465 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1798ec3af4194596838b8d95d1905524 in current term 2: Already voted for candidate b9b038b3d62a402a8acee576071f7fa6 in this term.
I20260620 23:02:16.398860 15140 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1798ec3af4194596838b8d95d1905524; no voters: a6d18b29646f40ac885810025578f279, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6
I20260620 23:02:16.399221 15578 raft_consensus.cc:2726] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 23:02:16.399257 15399 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.399972 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:16.400197 15465 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aaeacf1e4563412da1e4d6423583dc62 in current term 2: Already voted for candidate b9b038b3d62a402a8acee576071f7fa6 in this term.
I20260620 23:02:16.402616 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:16.402864 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1798ec3af4194596838b8d95d1905524 in term 1.
I20260620 23:02:16.403337 15579 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:16.410377 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463"
I20260620 23:02:16.410610 14945 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 1 FOLLOWER]: Advancing to term 2
W20260620 23:02:16.410820 14878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.413792 14944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:16.414412 14944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463"
I20260620 23:02:16.414927 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1798ec3af4194596838b8d95d1905524 in term 2.
I20260620 23:02:16.415453 15401 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6; no voters: 1798ec3af4194596838b8d95d1905524, a6d18b29646f40ac885810025578f279, d18d528e165543558ff363a3dc7f7463
I20260620 23:02:16.415779 15577 raft_consensus.cc:2726] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 23:02:16.417244 14878 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:16.419252 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "d18d528e165543558ff363a3dc7f7463" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "aaeacf1e4563412da1e4d6423583dc62" is_pre_election: true
I20260620 23:02:16.419601 15075 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d18d528e165543558ff363a3dc7f7463 in current term 2: Already voted for candidate b9b038b3d62a402a8acee576071f7fa6 in this term.
I20260620 23:02:16.420307 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:16.420591 14945 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aaeacf1e4563412da1e4d6423583dc62 in current term 2: Already voted for candidate 1798ec3af4194596838b8d95d1905524 in this term.
I20260620 23:02:16.421011 15011 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, aaeacf1e4563412da1e4d6423583dc62; no voters: a6d18b29646f40ac885810025578f279, b9b038b3d62a402a8acee576071f7fa6, d18d528e165543558ff363a3dc7f7463
I20260620 23:02:16.421412 15576 raft_consensus.cc:2726] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:02:16.422166 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "d18d528e165543558ff363a3dc7f7463" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:16.422432 15465 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d18d528e165543558ff363a3dc7f7463 in current term 2: Already voted for candidate b9b038b3d62a402a8acee576071f7fa6 in this term.
I20260620 23:02:16.422883 14879 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d18d528e165543558ff363a3dc7f7463; no voters: a6d18b29646f40ac885810025578f279, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6
I20260620 23:02:16.423331 15579 raft_consensus.cc:2726] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:02:16.424103 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "d18d528e165543558ff363a3dc7f7463" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:16.424351 15205 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d18d528e165543558ff363a3dc7f7463 in current term 2: Already voted for candidate 1798ec3af4194596838b8d95d1905524 in this term.
I20260620 23:02:16.471552 10778 test_util.cc:268] Using random seed: 1720459204
I20260620 23:02:17.079676 10778 test_util.cc:268] Using random seed: 1721067361
I20260620 23:02:17.787779 10778 test_util.cc:268] Using random seed: 1721775447
I20260620 23:02:17.978068 15611 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:17.978278 15611 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:17.979106 15611 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:17.979694 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:17.979737 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:17.979964 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 2.
I20260620 23:02:17.980050 15205 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 2.
I20260620 23:02:17.980329 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:17.980499 15011 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, aaeacf1e4563412da1e4d6423583dc62, d18d528e165543558ff363a3dc7f7463; no voters:
I20260620 23:02:17.980608 15465 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 2.
I20260620 23:02:17.980779 15611 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:02:17.980911 15611 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:17.981051 15611 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 2 FOLLOWER]: Advancing to term 3
W20260620 23:02:17.981691 15009 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:17.983326 15611 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:17.984064 15611 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 3 election: Requested vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:17.984491 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463"
I20260620 23:02:17.984504 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524"
I20260620 23:02:17.984715 15205 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:02:17.984720 14945 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:02:17.984737 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "aaeacf1e4563412da1e4d6423583dc62" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6"
I20260620 23:02:17.984931 15465 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 2 FOLLOWER]: Advancing to term 3
W20260620 23:02:17.985522 15009 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:17.986974 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 3.
I20260620 23:02:17.987455 15205 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 3.
I20260620 23:02:17.987519 15465 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaeacf1e4563412da1e4d6423583dc62 in term 3.
I20260620 23:02:17.987871 15011 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 4 yes votes; 1 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6, d18d528e165543558ff363a3dc7f7463; no voters: a6d18b29646f40ac885810025578f279
I20260620 23:02:17.988165 15611 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:02:17.988559 15611 raft_consensus.cc:686] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 3 LEADER]: Becoming Leader. State: Replica: aaeacf1e4563412da1e4d6423583dc62, State: Running, Role: LEADER
I20260620 23:02:17.988986 15611 consensus_queue.cc:237] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:17.992421 14801 catalog_manager.cc:5495] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 reported cstate change: term changed from 1 to 3, leader changed from a6d18b29646f40ac885810025578f279 (127.10.134.132) to aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130). New cstate: current_term: 3 leader_uuid: "aaeacf1e4563412da1e4d6423583dc62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:18.401000 15465 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Refusing update from remote peer aaeacf1e4563412da1e4d6423583dc62: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 23:02:18.401633 15611 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:02:18.526958 14945 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Refusing update from remote peer aaeacf1e4563412da1e4d6423583dc62: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 23:02:18.527624 15611 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:02:18.548363 15205 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Refusing update from remote peer aaeacf1e4563412da1e4d6423583dc62: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20260620 23:02:18.549495 15629 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 23:02:18.551123 15009 consensus_peers.cc:487] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 -> Peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Couldn't send request to peer a6d18b29646f40ac885810025578f279. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:18.596138 10778 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20260620 23:02:18.599838 15514 meta_cache.cc:302] tablet 006a43f02e7c4304a86b1a72aa1ab5e5: replica a6d18b29646f40ac885810025578f279 (127.10.134.132:43969) has failed: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
W20260620 23:02:18.610682 15166 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52988: Illegal state: replica 1798ec3af4194596838b8d95d1905524 is not leader of this config: current role FOLLOWER
I20260620 23:02:18.658320 10778 raft_consensus_election-itest.cc:408] Killing current leader aaeacf1e4563412da1e4d6423583dc62...
I20260620 23:02:18.658465 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 14994
I20260620 23:02:18.659409 15116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:18.755708 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 14994
I20260620 23:02:18.773352 10778 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260620 23:02:18.776212 10778 test_util.cc:268] Using random seed: 1722763885
I20260620 23:02:18.883565 10778 test_util.cc:268] Using random seed: 1722871247
I20260620 23:02:19.090767 10778 test_util.cc:268] Using random seed: 1723078424
I20260620 23:02:19.397992 10778 test_util.cc:268] Using random seed: 1723385637
I20260620 23:02:19.805523 10778 test_util.cc:268] Using random seed: 1723793172
I20260620 23:02:20.158465 15646 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aaeacf1e4563412da1e4d6423583dc62)
I20260620 23:02:20.158707 15646 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:20.159197 15647 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aaeacf1e4563412da1e4d6423583dc62)
I20260620 23:02:20.159371 15647 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:20.159605 15646 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377)
I20260620 23:02:20.160185 15649 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aaeacf1e4563412da1e4d6423583dc62)
I20260620 23:02:20.160208 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:20.160441 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 3.
I20260620 23:02:20.160400 15649 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:20.161003 15647 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:20.161337 15206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:20.161546 15206 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 3.
I20260620 23:02:20.162009 15401 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, b9b038b3d62a402a8acee576071f7fa6, d18d528e165543558ff363a3dc7f7463; no voters:
I20260620 23:02:20.162104 15465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "d18d528e165543558ff363a3dc7f7463" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:20.162386 15646 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260620 23:02:20.162520 15646 raft_consensus.cc:482] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader aaeacf1e4563412da1e4d6423583dc62)
I20260620 23:02:20.162571 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "d18d528e165543558ff363a3dc7f7463" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524" is_pre_election: true
I20260620 23:02:20.162695 15646 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 3 FOLLOWER]: Advancing to term 4
I20260620 23:02:20.162844 15205 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d18d528e165543558ff363a3dc7f7463 in term 3.
I20260620 23:02:20.167608 15646 raft_consensus.cc:504] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:20.167814 15649 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377), b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093)
I20260620 23:02:20.168295 15466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "b9b038b3d62a402a8acee576071f7fa6" is_pre_election: true
I20260620 23:02:20.168367 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "1798ec3af4194596838b8d95d1905524" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463" is_pre_election: true
I20260620 23:02:20.168594 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1798ec3af4194596838b8d95d1905524 in term 3.
I20260620 23:02:20.169195 15465 raft_consensus.cc:2379] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d18d528e165543558ff363a3dc7f7463 in current term 4: Already voted for candidate b9b038b3d62a402a8acee576071f7fa6 in this term.
I20260620 23:02:20.171128 15205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1798ec3af4194596838b8d95d1905524"
I20260620 23:02:20.171367 15205 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 3 FOLLOWER]: Advancing to term 4
I20260620 23:02:20.171439 15646 leader_election.cc:288] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 election: Requested vote from peers a6d18b29646f40ac885810025578f279 (127.10.134.132:43969), 1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013), aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759), d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377)
I20260620 23:02:20.172307 14945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "006a43f02e7c4304a86b1a72aa1ab5e5" candidate_uuid: "b9b038b3d62a402a8acee576071f7fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "d18d528e165543558ff363a3dc7f7463"
I20260620 23:02:20.172523 14945 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 3 FOLLOWER]: Advancing to term 4
W20260620 23:02:20.173339 14878 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
W20260620 23:02:20.173472 14878 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759): Network error: Client connection negotiation failed: client connection to 127.10.134.130:37759: connect: Connection refused (error 111)
I20260620 23:02:20.173558 14878 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, d18d528e165543558ff363a3dc7f7463; no voters: a6d18b29646f40ac885810025578f279, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6
W20260620 23:02:20.175341 15399 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:20.176208 14945 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 4.
I20260620 23:02:20.176496 15647 raft_consensus.cc:2726] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:20.176813 15205 raft_consensus.cc:2454] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b038b3d62a402a8acee576071f7fa6 in term 4.
I20260620 23:02:20.177311 15401 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, b9b038b3d62a402a8acee576071f7fa6, d18d528e165543558ff363a3dc7f7463; no voters:
I20260620 23:02:20.177587 15646 raft_consensus.cc:2781] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 4 FOLLOWER]: Leader election won for term 4
W20260620 23:02:20.178012 15399 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759): Network error: Client connection negotiation failed: client connection to 127.10.134.130:37759: connect: Connection refused (error 111)
I20260620 23:02:20.178311 15646 raft_consensus.cc:686] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [term 4 LEADER]: Becoming Leader. State: Replica: b9b038b3d62a402a8acee576071f7fa6, State: Running, Role: LEADER
W20260620 23:02:20.178851 15139 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
I20260620 23:02:20.178830 15646 consensus_queue.cc:237] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
W20260620 23:02:20.179090 15399 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111)
W20260620 23:02:20.179396 15139 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759): Network error: Client connection negotiation failed: client connection to 127.10.134.130:37759: connect: Connection refused (error 111)
W20260620 23:02:20.179396 15399 leader_election.cc:334] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759): Network error: Client connection negotiation failed: client connection to 127.10.134.130:37759: connect: Connection refused (error 111)
I20260620 23:02:20.179548 15139 leader_election.cc:302] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1798ec3af4194596838b8d95d1905524, d18d528e165543558ff363a3dc7f7463; no voters: a6d18b29646f40ac885810025578f279, aaeacf1e4563412da1e4d6423583dc62, b9b038b3d62a402a8acee576071f7fa6
I20260620 23:02:20.179867 15649 raft_consensus.cc:2726] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260620 23:02:20.182792 14802 catalog_manager.cc:5495] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 reported cstate change: term changed from 3 to 4, leader changed from aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130) to b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133). New cstate: current_term: 4 leader_uuid: "b9b038b3d62a402a8acee576071f7fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:20.313922 10778 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20260620 23:02:20.317591 15514 meta_cache.cc:302] tablet 006a43f02e7c4304a86b1a72aa1ab5e5: replica aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759) has failed: Network error: Client connection negotiation failed: client connection to 127.10.134.130:37759: connect: Connection refused (error 111)
W20260620 23:02:20.319936 15166 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52988: Illegal state: replica 1798ec3af4194596838b8d95d1905524 is not leader of this config: current role FOLLOWER
W20260620 23:02:20.333303 14905 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49098: Illegal state: replica d18d528e165543558ff363a3dc7f7463 is not leader of this config: current role FOLLOWER
I20260620 23:02:20.352097 14945 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P d18d528e165543558ff363a3dc7f7463 [term 4 FOLLOWER]: Refusing update from remote peer b9b038b3d62a402a8acee576071f7fa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20260620 23:02:20.352428 15205 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P 1798ec3af4194596838b8d95d1905524 [term 4 FOLLOWER]: Refusing update from remote peer b9b038b3d62a402a8acee576071f7fa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20260620 23:02:20.352995 15658 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 23:02:20.353451 15399 consensus_peers.cc:487] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 -> Peer a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): Couldn't send request to peer a6d18b29646f40ac885810025578f279. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.132:43969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:20.353451 15646 consensus_queue.cc:1035] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 23:02:20.353685 15399 consensus_peers.cc:487] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 -> Peer aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759): Couldn't send request to peer aaeacf1e4563412da1e4d6423583dc62. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.130:37759: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:20.360385 15668 mvcc.cc:205] Tried to move back new op lower bound from 7299057829275136000 to 7299057828578164736. Current Snapshot: MvccSnapshot[applied={T|T < 7299057829275136000}]
I20260620 23:02:20.395839 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:37759
--local_ip_for_outbound_sockets=127.10.134.130
--tserver_master_addrs=127.10.134.190:44137
--webserver_port=44611
--webserver_interface=127.10.134.130
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260620 23:02:20.720646 15679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:20.720907 15679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:20.720985 15679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:20.728031 15679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:20.728191 15679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:20.736835 15679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:37759
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=44611
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:20.738101 15679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:20.739835 15679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:20.752714 15684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:20.752799 15685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:20.752945 15687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:20.752862 15679 server_base.cc:1034] running on GCE node
I20260620 23:02:20.753816 15679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:20.754760 15679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:20.755963 15679 hybrid_clock.cc:648] HybridClock initialized: now 1781996540755904 us; error 34 us; skew 500 ppm
I20260620 23:02:20.756271 15679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:20.758658 15679 webserver.cc:460] Webserver started at http://127.10.134.130:44611/ using document root <none> and password file <none>
I20260620 23:02:20.759310 15679 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:20.759455 15679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:20.764369 15679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:02:20.767135 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:20.768527 15679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:20.768693 15679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aaeacf1e4563412da1e4d6423583dc62"
format_stamp: "Formatted at 2026-06-20 23:02:12 on dist-test-slave-8k25"
I20260620 23:02:20.769641 15679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:20.789640 15679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:20.790477 15679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:20.790756 15679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:20.791692 15679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:20.793447 15700 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:02:20.798674 15679 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:02:20.798795 15679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 23:02:20.798931 15679 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:02:20.801956 15679 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:02:20.802078 15679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.003s
I20260620 23:02:20.802392 15700 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Bootstrap starting.
I20260620 23:02:20.826849 15700 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:20.851990 15700 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:02:20.852640 15700 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Bootstrap complete.
I20260620 23:02:20.853128 15679 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:37759
I20260620 23:02:20.853168 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:37759 every 8 connection(s)
I20260620 23:02:20.853909 15700 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Time spent bootstrapping tablet: real 0.052s user 0.038s sys 0.011s
I20260620 23:02:20.854454 15679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:20.859555 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 15679
I20260620 23:02:20.860222 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.132:43969
--local_ip_for_outbound_sockets=127.10.134.132
--tserver_master_addrs=127.10.134.190:44137
--webserver_port=41299
--webserver_interface=127.10.134.132
--builtin_ntp_servers=127.10.134.148:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260620 23:02:20.862969 15700 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:20.864632 15700 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaeacf1e4563412da1e4d6423583dc62, State: Initialized, Role: FOLLOWER
I20260620 23:02:20.865456 15700 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:20.871285 15808 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:20.871701 15808 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:20.872771 15808 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:20.875259 15700 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62: Time spent starting tablet: real 0.021s user 0.005s sys 0.011s
I20260620 23:02:20.875905 14803 ts_manager.cc:194] Re-registered known tserver with Master: aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759)
I20260620 23:02:20.877677 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:54253
I20260620 23:02:20.880244 15808 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
W20260620 23:02:21.182183 15812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:21.182433 15812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:21.182511 15812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:21.189575 15812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:21.189733 15812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.132
I20260620 23:02:21.198434 15812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.132:43969
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.134.132
--webserver_port=41299
--tserver_master_addrs=127.10.134.190:44137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.132
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:21.199594 15812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:21.201397 15812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:21.214205 15818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:21.214294 15821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:21.214316 15819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:21.215169 15812 server_base.cc:1034] running on GCE node
I20260620 23:02:21.215760 15812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:21.216627 15812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:21.217862 15812 hybrid_clock.cc:648] HybridClock initialized: now 1781996541217806 us; error 45 us; skew 500 ppm
I20260620 23:02:21.218161 15812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:21.220340 15812 webserver.cc:460] Webserver started at http://127.10.134.132:41299/ using document root <none> and password file <none>
I20260620 23:02:21.221037 15812 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:21.221204 15812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:21.226050 15812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:21.228631 15827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:21.229770 15812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:21.229969 15812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a6d18b29646f40ac885810025578f279"
format_stamp: "Formatted at 2026-06-20 23:02:13 on dist-test-slave-8k25"
I20260620 23:02:21.230880 15812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:21.244771 15812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:21.245534 15812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:21.245800 15812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:21.246749 15812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:21.248477 15834 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:02:21.253722 15812 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:02:21.253849 15812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 23:02:21.254030 15812 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:02:21.257150 15812 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:02:21.257251 15812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260620 23:02:21.257599 15834 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Bootstrap starting.
I20260620 23:02:21.279726 15834 log.cc:824] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:21.297437 15812 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.132:43969
I20260620 23:02:21.297454 15941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.132:43969 every 8 connection(s)
I20260620 23:02:21.298938 15812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260620 23:02:21.302666 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 15812
I20260620 23:02:21.304731 15834 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:02:21.305421 15834 tablet_bootstrap.cc:492] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Bootstrap complete.
I20260620 23:02:21.306874 15834 ts_tablet_manager.cc:1397] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Time spent bootstrapping tablet: real 0.050s user 0.036s sys 0.012s
I20260620 23:02:21.314644 15942 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44137
I20260620 23:02:21.314939 15942 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:21.314981 15834 raft_consensus.cc:348] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:21.315907 15942 heartbeater.cc:511] Master 127.10.134.190:44137 requested a full tablet report, sending...
I20260620 23:02:21.315980 15834 raft_consensus.cc:729] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d18b29646f40ac885810025578f279, State: Initialized, Role: FOLLOWER
I20260620 23:02:21.316653 15834 consensus_queue.cc:260] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 } } peers { permanent_uuid: "1798ec3af4194596838b8d95d1905524" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 41013 } } peers { permanent_uuid: "aaeacf1e4563412da1e4d6423583dc62" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 37759 } } peers { permanent_uuid: "d18d528e165543558ff363a3dc7f7463" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 35377 } } peers { permanent_uuid: "b9b038b3d62a402a8acee576071f7fa6" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 38093 } }
I20260620 23:02:21.318442 15834 ts_tablet_manager.cc:1428] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260620 23:02:21.320199 14803 ts_manager.cc:194] Re-registered known tserver with Master: a6d18b29646f40ac885810025578f279 (127.10.134.132:43969)
I20260620 23:02:21.321283 14803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.132:58813
I20260620 23:02:21.323436 15942 heartbeater.cc:503] Master 127.10.134.190:44137 was elected leader, sending a full tablet report...
I20260620 23:02:21.415081 15186 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.418797 14925 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.423745 15446 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260620 23:02:21.425598 15876 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.429371 15742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.465431 15762 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P aaeacf1e4563412da1e4d6423583dc62 [term 3 FOLLOWER]: Advancing to term 4
I20260620 23:02:21.484436 15978 mvcc.cc:205] Tried to move back new op lower bound from 7299057829275136000 to 7299057828578164736. Current Snapshot: MvccSnapshot[applied={T|T < 7299057829275136000}]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0603d265b06b4184be4a1a58d2ce603b | 127.10.134.190:44137 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:42307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
1798ec3af4194596838b8d95d1905524 | 127.10.134.131:41013 | HEALTHY | <none> | 0 | 0
a6d18b29646f40ac885810025578f279 | 127.10.134.132:43969 | HEALTHY | <none> | 0 | 0
aaeacf1e4563412da1e4d6423583dc62 | 127.10.134.130:37759 | HEALTHY | <none> | 0 | 0
b9b038b3d62a402a8acee576071f7fa6 | 127.10.134.133:38093 | HEALTHY | <none> | 1 | 0
d18d528e165543558ff363a3dc7f7463 | 127.10.134.129:35377 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:35377
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:37759
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:41013
local_ip_for_outbound_sockets | 127.10.134.132 | experimental | 127.10.134.132:43969
local_ip_for_outbound_sockets | 127.10.134.133 | experimental | 127.10.134.133:38093
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:35377
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:37759
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:41013
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.134.132:43969
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.134.133:38093
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:42307 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet 006a43f02e7c4304a86b1a72aa1ab5e5 of table 'TestTable' is conflicted: 2 replicas' active configs disagree with the leader master's
a6d18b29646f40ac885810025578f279 (127.10.134.132:43969): RUNNING
1798ec3af4194596838b8d95d1905524 (127.10.134.131:41013): RUNNING
aaeacf1e4563412da1e4d6423583dc62 (127.10.134.130:37759): RUNNING
d18d528e165543558ff363a3dc7f7463 (127.10.134.129:35377): RUNNING
b9b038b3d62a402a8acee576071f7fa6 (127.10.134.133:38093): RUNNING [LEADER]
All reported replicas are:
A = a6d18b29646f40ac885810025578f279
B = 1798ec3af4194596838b8d95d1905524
C = aaeacf1e4563412da1e4d6423583dc62
D = d18d528e165543558ff363a3dc7f7463
E = b9b038b3d62a402a8acee576071f7fa6
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D E* | | | Yes
A | A B C D E | 1 | -1 | Yes
B | A B C D E* | 4 | -1 | Yes
C | A B C D E | 3 | -1 | Yes
D | A B C D E* | 4 | -1 | Yes
E | A B C D E* | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260620 23:02:21.496974 10778 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260620 23:02:21.574175 15897 raft_consensus.cc:3037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 1 FOLLOWER]: Advancing to term 4
I20260620 23:02:21.578518 15897 raft_consensus.cc:1261] T 006a43f02e7c4304a86b1a72aa1ab5e5 P a6d18b29646f40ac885810025578f279 [term 4 FOLLOWER]: Refusing update from remote peer b9b038b3d62a402a8acee576071f7fa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20260620 23:02:21.579396 15646 consensus_queue.cc:1037] T 006a43f02e7c4304a86b1a72aa1ab5e5 P b9b038b3d62a402a8acee576071f7fa6 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "a6d18b29646f40ac885810025578f279" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 43969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:02:21.586622 15983 mvcc.cc:205] Tried to move back new op lower bound from 7299057822202753024 to 7299057819607285760. Current Snapshot: MvccSnapshot[applied={T|T < 7299057822202753024}]
I20260620 23:02:21.720973 15186 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.723495 15876 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.729453 15742 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.744521 14925 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260620 23:02:21.755465 15446 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
0603d265b06b4184be4a1a58d2ce603b | 127.10.134.190:44137 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:42307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
1798ec3af4194596838b8d95d1905524 | 127.10.134.131:41013 | HEALTHY | <none> | 0 | 0
a6d18b29646f40ac885810025578f279 | 127.10.134.132:43969 | HEALTHY | <none> | 0 | 0
aaeacf1e4563412da1e4d6423583dc62 | 127.10.134.130:37759 | HEALTHY | <none> | 0 | 0
b9b038b3d62a402a8acee576071f7fa6 | 127.10.134.133:38093 | HEALTHY | <none> | 1 | 0
d18d528e165543558ff363a3dc7f7463 | 127.10.134.129:35377 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.10.134.129 | experimental | 127.10.134.129:35377
local_ip_for_outbound_sockets | 127.10.134.130 | experimental | 127.10.134.130:37759
local_ip_for_outbound_sockets | 127.10.134.131 | experimental | 127.10.134.131:41013
local_ip_for_outbound_sockets | 127.10.134.132 | experimental | 127.10.134.132:43969
local_ip_for_outbound_sockets | 127.10.134.133 | experimental | 127.10.134.133:38093
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.10.134.129:35377
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.10.134.130:37759
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.10.134.131:41013
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.10.134.132:43969
server_dump_info_path | /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.10.134.133:38093
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.10.134.148:42307 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260620 23:02:21.798346 10778 log_verifier.cc:126] Checking tablet 006a43f02e7c4304a86b1a72aa1ab5e5
I20260620 23:02:21.856837 10778 log_verifier.cc:177] Verified matching terms for 18 ops in tablet 006a43f02e7c4304a86b1a72aa1ab5e5
I20260620 23:02:21.967427 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 14863
I20260620 23:02:21.968502 14985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:22.067078 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 14863
I20260620 23:02:22.087317 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 15679
I20260620 23:02:22.094578 15803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:22.186167 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 15679
I20260620 23:02:22.202987 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 15124
I20260620 23:02:22.204272 15246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:22.299748 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 15124
I20260620 23:02:22.317358 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 15812
I20260620 23:02:22.323465 15936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:22.415104 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 15812
I20260620 23:02:22.431756 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 15384
I20260620 23:02:22.432844 15505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:22.528460 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 15384
I20260620 23:02:22.548498 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 14773
I20260620 23:02:22.549605 14834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:22.657764 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 14773
2026-06-20T23:02:22Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (11022 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20260620 23:02:22.704212 10778 test_util.cc:268] Using random seed: 1726691900
I20260620 23:02:22.705832 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:22.705891 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:22.705971 10778 ts_itest-base.cc:114] 1 tablet servers
I20260620 23:02:22.706014 10778 ts_itest-base.cc:115] 1 replicas per TS
I20260620 23:02:22.706066 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:22Z Disabled control of system clock
I20260620 23:02:22.734287 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:35771
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:35771 with env {}
W20260620 23:02:23.051901 16040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:23.052250 16040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:23.052392 16040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:23.059432 16040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:23.059569 16040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:23.059664 16040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:23.059765 16040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:23.068684 16040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:35771
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:23.070053 16040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:23.071799 16040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:23.083531 16046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.083593 16045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.083595 16048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:23.084520 16040 server_base.cc:1034] running on GCE node
I20260620 23:02:23.085321 16040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:23.086825 16040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:23.088071 16040 hybrid_clock.cc:648] HybridClock initialized: now 1781996543088013 us; error 30 us; skew 500 ppm
I20260620 23:02:23.088413 16040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:23.093694 16040 webserver.cc:460] Webserver started at http://127.10.134.190:45821/ using document root <none> and password file <none>
I20260620 23:02:23.094441 16040 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:23.094586 16040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:23.094893 16040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:23.097460 16040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2e70313b7f484ea6974b546cc4ec05a9"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-8k25"
I20260620 23:02:23.098255 16040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2e70313b7f484ea6974b546cc4ec05a9"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-8k25"
I20260620 23:02:23.103091 16040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:02:23.106526 16054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.107764 16040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:23.107936 16040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2e70313b7f484ea6974b546cc4ec05a9"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-8k25"
I20260620 23:02:23.108116 16040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:23.132164 16040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:23.133002 16040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:23.133312 16040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:23.151319 16040 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:35771
I20260620 23:02:23.151348 16105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:35771 every 8 connection(s)
I20260620 23:02:23.152699 16040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:23.156479 16106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:23.162890 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16040
I20260620 23:02:23.163061 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:23.166816 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9: Bootstrap starting.
I20260620 23:02:23.170185 16106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:23.171264 16106 log.cc:824] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:23.174201 16106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9: No bootstrap required, opened a new log
I20260620 23:02:23.182463 16106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70313b7f484ea6974b546cc4ec05a9" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35771 } }
I20260620 23:02:23.182832 16106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:23.182953 16106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e70313b7f484ea6974b546cc4ec05a9, State: Initialized, Role: FOLLOWER
I20260620 23:02:23.183521 16106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70313b7f484ea6974b546cc4ec05a9" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35771 } }
I20260620 23:02:23.183745 16106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:23.183872 16106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:23.184047 16106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:23.186080 16106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70313b7f484ea6974b546cc4ec05a9" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35771 } }
I20260620 23:02:23.186546 16106 leader_election.cc:302] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e70313b7f484ea6974b546cc4ec05a9; no voters:
I20260620 23:02:23.187096 16106 leader_election.cc:288] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:23.187219 16111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:23.187920 16111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [term 1 LEADER]: Becoming Leader. State: Replica: 2e70313b7f484ea6974b546cc4ec05a9, State: Running, Role: LEADER
I20260620 23:02:23.188390 16111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70313b7f484ea6974b546cc4ec05a9" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35771 } }
I20260620 23:02:23.189074 16106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:23.191162 16113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e70313b7f484ea6974b546cc4ec05a9. Latest consensus state: current_term: 1 leader_uuid: "2e70313b7f484ea6974b546cc4ec05a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70313b7f484ea6974b546cc4ec05a9" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35771 } } }
I20260620 23:02:23.191835 16113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:23.191277 16112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e70313b7f484ea6974b546cc4ec05a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70313b7f484ea6974b546cc4ec05a9" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35771 } } }
I20260620 23:02:23.192833 16112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:23.193393 16120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:23.198551 16120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:23.207348 16120 catalog_manager.cc:1347] Generated new cluster ID: 39f45adedbf84a6084a5a8a587230a44
I20260620 23:02:23.207481 16120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:23.224553 16120 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:23.225723 16120 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:23.238092 16120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e70313b7f484ea6974b546cc4ec05a9: Generated new TSK 0
I20260620 23:02:23.238931 16120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:23.245318 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35771
--builtin_ntp_servers=127.10.134.148:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260620 23:02:23.570179 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:23.570449 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:23.570608 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:23.577847 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:23.578110 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:23.587188 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35771
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:23.588508 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:23.590340 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:23.602986 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.603024 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.603027 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:23.603796 16130 server_base.cc:1034] running on GCE node
I20260620 23:02:23.604396 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:23.605329 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:23.606556 16130 hybrid_clock.cc:648] HybridClock initialized: now 1781996543606498 us; error 59 us; skew 500 ppm
I20260620 23:02:23.606859 16130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:23.609026 16130 webserver.cc:460] Webserver started at http://127.10.134.129:41453/ using document root <none> and password file <none>
I20260620 23:02:23.609663 16130 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:23.609813 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:23.610147 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:23.612591 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cf16c6abb3b14fe7a10fb3445092db77"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-8k25"
I20260620 23:02:23.613332 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cf16c6abb3b14fe7a10fb3445092db77"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-8k25"
I20260620 23:02:23.618080 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:02:23.621279 16144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.622504 16130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:23.622676 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf16c6abb3b14fe7a10fb3445092db77"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-8k25"
I20260620 23:02:23.622874 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:23.638000 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:23.638744 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:23.639055 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:23.639907 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:23.641278 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:23.641381 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.641534 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:23.641606 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.673743 16130 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:38985
I20260620 23:02:23.673790 16256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:38985 every 8 connection(s)
I20260620 23:02:23.675056 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:23.677855 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16130
I20260620 23:02:23.678082 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:23.689196 16257 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35771
I20260620 23:02:23.689474 16257 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:23.690234 16257 heartbeater.cc:511] Master 127.10.134.190:35771 requested a full tablet report, sending...
I20260620 23:02:23.692433 16071 ts_manager.cc:194] Registered new tserver with Master: cf16c6abb3b14fe7a10fb3445092db77 (127.10.134.129:38985)
I20260620 23:02:23.694434 10778 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260620 23:02:23.694473 16071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:45073
I20260620 23:02:23.711478 16071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59458:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260620 23:02:23.739228 16192 tablet_service.cc:1467] Processing CreateTablet for tablet 1e397406a4a3484d8edd936c8e944e61 (DEFAULT_TABLE table=TestTable [id=c877053a8f85404790ea5dd5412a6e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:23.740643 16192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e397406a4a3484d8edd936c8e944e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:23.750553 16268 tablet_bootstrap.cc:492] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77: Bootstrap starting.
I20260620 23:02:23.753340 16268 tablet_bootstrap.cc:654] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:23.754384 16268 log.cc:824] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:23.756642 16268 tablet_bootstrap.cc:492] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77: No bootstrap required, opened a new log
I20260620 23:02:23.756896 16268 ts_tablet_manager.cc:1397] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260620 23:02:23.763861 16268 raft_consensus.cc:348] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf16c6abb3b14fe7a10fb3445092db77" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38985 } }
I20260620 23:02:23.764230 16268 raft_consensus.cc:374] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:23.764346 16268 raft_consensus.cc:729] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf16c6abb3b14fe7a10fb3445092db77, State: Initialized, Role: FOLLOWER
I20260620 23:02:23.764957 16268 consensus_queue.cc:260] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf16c6abb3b14fe7a10fb3445092db77" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38985 } }
I20260620 23:02:23.765197 16268 raft_consensus.cc:388] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:23.765347 16268 raft_consensus.cc:482] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:23.765511 16268 raft_consensus.cc:3037] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:23.767889 16268 raft_consensus.cc:504] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf16c6abb3b14fe7a10fb3445092db77" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38985 } }
I20260620 23:02:23.768374 16268 leader_election.cc:302] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf16c6abb3b14fe7a10fb3445092db77; no voters:
I20260620 23:02:23.768960 16268 leader_election.cc:288] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:23.769116 16270 raft_consensus.cc:2781] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:23.769852 16270 raft_consensus.cc:686] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [term 1 LEADER]: Becoming Leader. State: Replica: cf16c6abb3b14fe7a10fb3445092db77, State: Running, Role: LEADER
I20260620 23:02:23.770066 16257 heartbeater.cc:503] Master 127.10.134.190:35771 was elected leader, sending a full tablet report...
I20260620 23:02:23.770499 16270 consensus_queue.cc:237] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf16c6abb3b14fe7a10fb3445092db77" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38985 } }
I20260620 23:02:23.770686 16268 ts_tablet_manager.cc:1428] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 23:02:23.775534 16071 catalog_manager.cc:5495] T 1e397406a4a3484d8edd936c8e944e61 P cf16c6abb3b14fe7a10fb3445092db77 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf16c6abb3b14fe7a10fb3445092db77 (127.10.134.129). New cstate: current_term: 1 leader_uuid: "cf16c6abb3b14fe7a10fb3445092db77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf16c6abb3b14fe7a10fb3445092db77" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38985 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:23.795042 10778 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260620 23:02:23.796805 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cf16c6abb3b14fe7a10fb3445092db77 to finish bootstrapping
I20260620 23:02:23.813836 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16130
I20260620 23:02:23.814819 16252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:23.903116 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16130
I20260620 23:02:23.919817 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16040
I20260620 23:02:23.920950 16101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:24.023365 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16040
2026-06-20T23:02:24Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (1356 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20260620 23:02:24.060938 10778 test_util.cc:268] Using random seed: 1728048613
I20260620 23:02:24.062613 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:24.062692 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:24.062763 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:02:24.062855 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:02:24.062922 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:24Z Disabled control of system clock
I20260620 23:02:24.091960 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:39079
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:39079
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260620 23:02:24.405827 16286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:24.406231 16286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:24.406368 16286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:24.413664 16286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:24.413748 16286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:24.413826 16286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:24.413925 16286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:24.422863 16286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:39079
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:24.424111 16286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:24.425835 16286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:24.437222 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.437326 16292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.437292 16294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:24.438143 16286 server_base.cc:1034] running on GCE node
I20260620 23:02:24.438892 16286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:24.440187 16286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:24.441421 16286 hybrid_clock.cc:648] HybridClock initialized: now 1781996544441356 us; error 55 us; skew 500 ppm
I20260620 23:02:24.441740 16286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:24.443907 16286 webserver.cc:460] Webserver started at http://127.10.134.190:35377/ using document root <none> and password file <none>
I20260620 23:02:24.444550 16286 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:24.444700 16286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:24.444988 16286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:24.447631 16286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "82c7806247a8411186ef5dc98ebb2d06"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-8k25"
I20260620 23:02:24.448359 16286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "82c7806247a8411186ef5dc98ebb2d06"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-8k25"
I20260620 23:02:24.453244 16286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:02:24.456630 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.458031 16286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:02:24.458194 16286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "82c7806247a8411186ef5dc98ebb2d06"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-8k25"
I20260620 23:02:24.458386 16286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:24.470883 16286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:24.471633 16286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:24.471956 16286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:24.489780 16286 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:39079
I20260620 23:02:24.489820 16351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:39079 every 8 connection(s)
I20260620 23:02:24.491084 16286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:24.494629 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:24.499715 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16286
I20260620 23:02:24.499925 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:24.504773 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06: Bootstrap starting.
I20260620 23:02:24.507929 16352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:24.509101 16352 log.cc:824] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:24.511730 16352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06: No bootstrap required, opened a new log
I20260620 23:02:24.519146 16352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7806247a8411186ef5dc98ebb2d06" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 39079 } }
I20260620 23:02:24.519454 16352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:24.519596 16352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c7806247a8411186ef5dc98ebb2d06, State: Initialized, Role: FOLLOWER
I20260620 23:02:24.520182 16352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7806247a8411186ef5dc98ebb2d06" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 39079 } }
I20260620 23:02:24.520393 16352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:24.520557 16352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:24.520722 16352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:24.522748 16352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7806247a8411186ef5dc98ebb2d06" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 39079 } }
I20260620 23:02:24.523200 16352 leader_election.cc:302] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82c7806247a8411186ef5dc98ebb2d06; no voters:
I20260620 23:02:24.523780 16352 leader_election.cc:288] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:24.523908 16357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:24.524612 16357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [term 1 LEADER]: Becoming Leader. State: Replica: 82c7806247a8411186ef5dc98ebb2d06, State: Running, Role: LEADER
I20260620 23:02:24.525151 16357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7806247a8411186ef5dc98ebb2d06" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 39079 } }
I20260620 23:02:24.525730 16352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:24.527997 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82c7806247a8411186ef5dc98ebb2d06. Latest consensus state: current_term: 1 leader_uuid: "82c7806247a8411186ef5dc98ebb2d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7806247a8411186ef5dc98ebb2d06" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 39079 } } }
I20260620 23:02:24.528420 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:24.528276 16358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82c7806247a8411186ef5dc98ebb2d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7806247a8411186ef5dc98ebb2d06" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 39079 } } }
I20260620 23:02:24.528589 16358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:24.530493 16366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:24.534662 16366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:24.546303 16366 catalog_manager.cc:1347] Generated new cluster ID: 814863ae061640c3af3afcea45d60d76
I20260620 23:02:24.546553 16366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:24.557703 16366 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:24.559564 16366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:24.570431 16366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82c7806247a8411186ef5dc98ebb2d06: Generated new TSK 0
I20260620 23:02:24.571450 16366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:24.579468 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:39079
--builtin_ntp_servers=127.10.134.148:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260620 23:02:24.898694 16376 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:24.899034 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:24.899147 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:24.899272 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:24.906354 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:24.906553 16376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:24.915328 16376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:39079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:24.916565 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:24.918277 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:24.931102 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.931150 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.931763 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:24.932091 16376 server_base.cc:1034] running on GCE node
I20260620 23:02:24.932683 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:24.933531 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:24.934718 16376 hybrid_clock.cc:648] HybridClock initialized: now 1781996544934662 us; error 52 us; skew 500 ppm
I20260620 23:02:24.935026 16376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:24.937194 16376 webserver.cc:460] Webserver started at http://127.10.134.129:39039/ using document root <none> and password file <none>
I20260620 23:02:24.937865 16376 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:24.938011 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:24.938321 16376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:24.940764 16376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2087053f4e7c45ad82dc7edef8ad4f0e"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-8k25"
I20260620 23:02:24.941515 16376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2087053f4e7c45ad82dc7edef8ad4f0e"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-8k25"
I20260620 23:02:24.946296 16376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260620 23:02:24.949548 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.950899 16376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:24.951059 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2087053f4e7c45ad82dc7edef8ad4f0e"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-8k25"
I20260620 23:02:24.951259 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:24.967084 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:24.967818 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:24.968062 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:24.968871 16376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:24.970256 16376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:24.970345 16376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.970424 16376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:24.970479 16376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.999845 16376 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:41837
I20260620 23:02:24.999917 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:41837 every 8 connection(s)
I20260620 23:02:25.001149 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:25.010988 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16376
I20260620 23:02:25.011155 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:25.014443 16503 heartbeater.cc:347] Connected to a master server at 127.10.134.190:39079
I20260620 23:02:25.014722 16503 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:25.014894 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:39079
--builtin_ntp_servers=127.10.134.148:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:25.015439 16503 heartbeater.cc:511] Master 127.10.134.190:39079 requested a full tablet report, sending...
I20260620 23:02:25.017482 16317 ts_manager.cc:194] Registered new tserver with Master: 2087053f4e7c45ad82dc7edef8ad4f0e (127.10.134.129:41837)
I20260620 23:02:25.019601 16317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:48787
W20260620 23:02:25.332304 16506 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:25.332665 16506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:25.332775 16506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:25.332908 16506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:25.340265 16506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:25.340490 16506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:25.349640 16506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:39079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:25.351002 16506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:25.352767 16506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:25.366111 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:25.366974 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:25.367311 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:25.375855 16506 server_base.cc:1034] running on GCE node
I20260620 23:02:25.376911 16506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:25.378378 16506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:25.379665 16506 hybrid_clock.cc:648] HybridClock initialized: now 1781996545379611 us; error 42 us; skew 500 ppm
I20260620 23:02:25.380121 16506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:25.383574 16506 webserver.cc:460] Webserver started at http://127.10.134.130:37441/ using document root <none> and password file <none>
I20260620 23:02:25.384552 16506 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:25.384675 16506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:25.385084 16506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:25.388948 16506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-8k25"
I20260620 23:02:25.390005 16506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-8k25"
I20260620 23:02:25.396822 16506 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:02:25.400393 16520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.402385 16506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:25.402598 16506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-8k25"
I20260620 23:02:25.402818 16506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:25.415795 16506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:25.416538 16506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:25.416821 16506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:25.417698 16506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:25.419109 16506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:25.419243 16506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.419371 16506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:25.419456 16506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.451463 16506 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:39241
I20260620 23:02:25.451505 16632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:39241 every 8 connection(s)
I20260620 23:02:25.452852 16506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:25.456514 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16506
I20260620 23:02:25.456713 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:25.459985 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:39079
--builtin_ntp_servers=127.10.134.148:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:25.471375 16633 heartbeater.cc:347] Connected to a master server at 127.10.134.190:39079
I20260620 23:02:25.471728 16633 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:25.472488 16633 heartbeater.cc:511] Master 127.10.134.190:39079 requested a full tablet report, sending...
I20260620 23:02:25.473997 16317 ts_manager.cc:194] Registered new tserver with Master: b6eda64bbd4f4eedaa3aa8f4d72bed95 (127.10.134.130:39241)
I20260620 23:02:25.475282 16317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:50541
W20260620 23:02:25.779806 16636 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:25.780084 16636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:25.780140 16636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:25.780210 16636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:25.787371 16636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:25.787518 16636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:25.796268 16636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:39079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:25.797485 16636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:25.799245 16636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:25.811990 16642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:25.812193 16644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:25.812199 16641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:25.813447 16636 server_base.cc:1034] running on GCE node
I20260620 23:02:25.814026 16636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:25.814900 16636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:25.816092 16636 hybrid_clock.cc:648] HybridClock initialized: now 1781996545816052 us; error 38 us; skew 500 ppm
I20260620 23:02:25.816362 16636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:25.818602 16636 webserver.cc:460] Webserver started at http://127.10.134.131:35993/ using document root <none> and password file <none>
I20260620 23:02:25.819233 16636 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:25.819317 16636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:25.819576 16636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:25.822161 16636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "67ce2804ff8e435c8b40b2f84235eb6b"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-8k25"
I20260620 23:02:25.822871 16636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "67ce2804ff8e435c8b40b2f84235eb6b"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-8k25"
I20260620 23:02:25.827898 16636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:02:25.831254 16650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.832592 16636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:25.832742 16636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "67ce2804ff8e435c8b40b2f84235eb6b"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-8k25"
I20260620 23:02:25.832953 16636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:25.864054 16636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:25.864923 16636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:25.865224 16636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:25.866257 16636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:25.867662 16636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:25.867769 16636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.867893 16636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:25.867980 16636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.899660 16636 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:42761
I20260620 23:02:25.899682 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:42761 every 8 connection(s)
I20260620 23:02:25.900964 16636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:25.903394 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16636
I20260620 23:02:25.903543 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:25.914795 16763 heartbeater.cc:347] Connected to a master server at 127.10.134.190:39079
I20260620 23:02:25.915062 16763 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:25.915659 16763 heartbeater.cc:511] Master 127.10.134.190:39079 requested a full tablet report, sending...
I20260620 23:02:25.916944 16317 ts_manager.cc:194] Registered new tserver with Master: 67ce2804ff8e435c8b40b2f84235eb6b (127.10.134.131:42761)
I20260620 23:02:25.917748 16317 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:38641
I20260620 23:02:25.919384 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:25.934834 16317 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38566:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260620 23:02:25.941447 16317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:25.966230 16438 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc5b46c51f42c8a7dc83a735d92721 (DEFAULT_TABLE table=TestTable [id=f3ac16d007454f9e8d39765482120efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:25.966322 16698 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc5b46c51f42c8a7dc83a735d92721 (DEFAULT_TABLE table=TestTable [id=f3ac16d007454f9e8d39765482120efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:25.967244 16568 tablet_service.cc:1467] Processing CreateTablet for tablet 31cc5b46c51f42c8a7dc83a735d92721 (DEFAULT_TABLE table=TestTable [id=f3ac16d007454f9e8d39765482120efa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:25.967680 16698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc5b46c51f42c8a7dc83a735d92721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:25.967681 16438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc5b46c51f42c8a7dc83a735d92721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:25.968273 16568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31cc5b46c51f42c8a7dc83a735d92721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:25.978992 16781 tablet_bootstrap.cc:492] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95: Bootstrap starting.
I20260620 23:02:25.981105 16782 tablet_bootstrap.cc:492] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e: Bootstrap starting.
I20260620 23:02:25.982565 16781 tablet_bootstrap.cc:654] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:25.982739 16783 tablet_bootstrap.cc:492] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b: Bootstrap starting.
I20260620 23:02:25.984408 16781 log.cc:824] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:25.984474 16782 tablet_bootstrap.cc:654] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:25.985526 16782 log.cc:824] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:25.985925 16783 tablet_bootstrap.cc:654] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:25.986992 16783 log.cc:824] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:25.987185 16781 tablet_bootstrap.cc:492] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95: No bootstrap required, opened a new log
I20260620 23:02:25.987463 16781 ts_tablet_manager.cc:1397] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:02:25.988216 16782 tablet_bootstrap.cc:492] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e: No bootstrap required, opened a new log
I20260620 23:02:25.988459 16782 ts_tablet_manager.cc:1397] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:02:25.989259 16783 tablet_bootstrap.cc:492] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b: No bootstrap required, opened a new log
I20260620 23:02:25.989496 16783 ts_tablet_manager.cc:1397] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:02:25.996387 16783 raft_consensus.cc:348] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:25.996745 16783 raft_consensus.cc:729] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ce2804ff8e435c8b40b2f84235eb6b, State: Initialized, Role: FOLLOWER
I20260620 23:02:25.997157 16783 consensus_queue.cc:260] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:25.997663 16781 raft_consensus.cc:348] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:25.998302 16763 heartbeater.cc:503] Master 127.10.134.190:39079 was elected leader, sending a full tablet report...
I20260620 23:02:25.998750 16783 ts_tablet_manager.cc:1428] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20260620 23:02:25.998566 16782 raft_consensus.cc:348] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:25.998075 16781 raft_consensus.cc:729] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6eda64bbd4f4eedaa3aa8f4d72bed95, State: Initialized, Role: FOLLOWER
I20260620 23:02:25.998950 16782 raft_consensus.cc:729] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2087053f4e7c45ad82dc7edef8ad4f0e, State: Initialized, Role: FOLLOWER
I20260620 23:02:25.999365 16782 consensus_queue.cc:260] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:25.999440 16781 consensus_queue.cc:260] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:26.005343 16633 heartbeater.cc:503] Master 127.10.134.190:39079 was elected leader, sending a full tablet report...
I20260620 23:02:26.005959 16781 ts_tablet_manager.cc:1428] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260620 23:02:26.007916 16503 heartbeater.cc:503] Master 127.10.134.190:39079 was elected leader, sending a full tablet report...
I20260620 23:02:26.008633 16782 ts_tablet_manager.cc:1428] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20260620 23:02:26.021860 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:26.023928 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2087053f4e7c45ad82dc7edef8ad4f0e to finish bootstrapping
I20260620 23:02:26.032459 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b6eda64bbd4f4eedaa3aa8f4d72bed95 to finish bootstrapping
I20260620 23:02:26.040114 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 67ce2804ff8e435c8b40b2f84235eb6b to finish bootstrapping
I20260620 23:02:26.055522 16458 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31cc5b46c51f42c8a7dc83a735d92721"
dest_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e"
from {username='slave'} at 127.0.0.1:39652
I20260620 23:02:26.055853 16458 raft_consensus.cc:482] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:26.056008 16458 raft_consensus.cc:3037] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:26.058008 16458 raft_consensus.cc:504] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:26.059327 16458 leader_election.cc:288] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [CANDIDATE]: Term 1 election: Requested vote from peers 67ce2804ff8e435c8b40b2f84235eb6b (127.10.134.131:42761), b6eda64bbd4f4eedaa3aa8f4d72bed95 (127.10.134.130:39241)
I20260620 23:02:26.067530 16718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc5b46c51f42c8a7dc83a735d92721" candidate_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67ce2804ff8e435c8b40b2f84235eb6b"
I20260620 23:02:26.067981 16718 raft_consensus.cc:3037] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:26.068332 16588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31cc5b46c51f42c8a7dc83a735d92721" candidate_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95"
I20260620 23:02:26.068737 16588 raft_consensus.cc:3037] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:26.071161 16718 raft_consensus.cc:2454] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2087053f4e7c45ad82dc7edef8ad4f0e in term 1.
I20260620 23:02:26.071686 16588 raft_consensus.cc:2454] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2087053f4e7c45ad82dc7edef8ad4f0e in term 1.
I20260620 23:02:26.071735 16394 leader_election.cc:302] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2087053f4e7c45ad82dc7edef8ad4f0e, 67ce2804ff8e435c8b40b2f84235eb6b; no voters:
I20260620 23:02:26.072175 16788 raft_consensus.cc:2781] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:26.072746 16788 raft_consensus.cc:686] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 LEADER]: Becoming Leader. State: Replica: 2087053f4e7c45ad82dc7edef8ad4f0e, State: Running, Role: LEADER
I20260620 23:02:26.073230 16788 consensus_queue.cc:237] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:26.077881 16316 catalog_manager.cc:5495] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2087053f4e7c45ad82dc7edef8ad4f0e (127.10.134.129). New cstate: current_term: 1 leader_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:26.088378 16718 raft_consensus.cc:1261] T 31cc5b46c51f42c8a7dc83a735d92721 P 67ce2804ff8e435c8b40b2f84235eb6b [term 1 FOLLOWER]: Refusing update from remote peer 2087053f4e7c45ad82dc7edef8ad4f0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:26.088378 16588 raft_consensus.cc:1261] T 31cc5b46c51f42c8a7dc83a735d92721 P b6eda64bbd4f4eedaa3aa8f4d72bed95 [term 1 FOLLOWER]: Refusing update from remote peer 2087053f4e7c45ad82dc7edef8ad4f0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:26.089243 16788 consensus_queue.cc:1035] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:26.089587 16794 consensus_queue.cc:1035] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:26.099923 16796 mvcc.cc:205] Tried to move back new op lower bound from 7299057852768133120 to 7299057852720562176. Current Snapshot: MvccSnapshot[applied={T|T < 7299057852768133120}]
I20260620 23:02:26.099929 16797 mvcc.cc:205] Tried to move back new op lower bound from 7299057852768133120 to 7299057852720562176. Current Snapshot: MvccSnapshot[applied={T|T < 7299057852768133120}]
I20260620 23:02:26.103628 16798 mvcc.cc:205] Tried to move back new op lower bound from 7299057852768133120 to 7299057852720562176. Current Snapshot: MvccSnapshot[applied={T|T < 7299057852768133120}]
I20260620 23:02:26.120088 16458 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31cc5b46c51f42c8a7dc83a735d92721"
dest_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e"
from {username='slave'} at 127.0.0.1:39652
I20260620 23:02:26.120368 16458 raft_consensus.cc:580] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 LEADER]: Received request to step down
I20260620 23:02:26.120448 16458 raft_consensus.cc:3032] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:26.120532 16458 raft_consensus.cc:729] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2087053f4e7c45ad82dc7edef8ad4f0e, State: Running, Role: LEADER
I20260620 23:02:26.120767 16458 consensus_queue.cc:260] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 41837 } } peers { permanent_uuid: "67ce2804ff8e435c8b40b2f84235eb6b" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42761 } } peers { permanent_uuid: "b6eda64bbd4f4eedaa3aa8f4d72bed95" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 39241 } }
I20260620 23:02:26.121263 16458 raft_consensus.cc:3037] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:26.123284 16458 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31cc5b46c51f42c8a7dc83a735d92721"
dest_uuid: "2087053f4e7c45ad82dc7edef8ad4f0e"
from {username='slave'} at 127.0.0.1:39652
I20260620 23:02:26.123495 16458 raft_consensus.cc:573] T 31cc5b46c51f42c8a7dc83a735d92721 P 2087053f4e7c45ad82dc7edef8ad4f0e [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20260620 23:02:26.125250 16418 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39652: Illegal state: replica 2087053f4e7c45ad82dc7edef8ad4f0e is not leader of this config: current role FOLLOWER
I20260620 23:02:26.126588 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16376
I20260620 23:02:26.128167 16498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:26.218291 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16376
I20260620 23:02:26.235576 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16506
I20260620 23:02:26.236688 16628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:26.324308 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16506
I20260620 23:02:26.341172 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16636
I20260620 23:02:26.342527 16758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:26.429563 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16636
I20260620 23:02:26.445537 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16286
I20260620 23:02:26.446796 16347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:26.553813 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16286
2026-06-20T23:02:26Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (2529 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20260620 23:02:26.590502 10778 test_util.cc:268] Using random seed: 1730578185
I20260620 23:02:26.592226 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:26.592317 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:26.592393 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:02:26.592443 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:02:26.592505 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:26Z Disabled control of system clock
I20260620 23:02:26.620635 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:35543
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:35543
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260620 23:02:26.941033 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:26.941388 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:26.941526 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:26.948702 16815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:26.948843 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:26.948935 16815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:26.949033 16815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:26.958101 16815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:35543
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:26.959388 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:26.961091 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:26.972536 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:26.972675 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:26.972824 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:26.974013 16815 server_base.cc:1034] running on GCE node
I20260620 23:02:26.974758 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:26.976080 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:26.977312 16815 hybrid_clock.cc:648] HybridClock initialized: now 1781996546977245 us; error 61 us; skew 500 ppm
I20260620 23:02:26.977636 16815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:26.980268 16815 webserver.cc:460] Webserver started at http://127.10.134.190:45465/ using document root <none> and password file <none>
I20260620 23:02:26.980939 16815 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:26.981082 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:26.981400 16815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:26.984002 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cb1c4a403d3c46698effde494c86dc1c"
format_stamp: "Formatted at 2026-06-20 23:02:26 on dist-test-slave-8k25"
I20260620 23:02:26.984745 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cb1c4a403d3c46698effde494c86dc1c"
format_stamp: "Formatted at 2026-06-20 23:02:26 on dist-test-slave-8k25"
I20260620 23:02:26.989885 16815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:02:26.993428 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:26.994982 16815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:26.995153 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cb1c4a403d3c46698effde494c86dc1c"
format_stamp: "Formatted at 2026-06-20 23:02:26 on dist-test-slave-8k25"
I20260620 23:02:26.995357 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:27.012058 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:27.012805 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:27.013098 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:27.030472 16815 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:35543
I20260620 23:02:27.030501 16880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:35543 every 8 connection(s)
I20260620 23:02:27.031778 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:27.035454 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:27.038245 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16815
I20260620 23:02:27.038446 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:27.046693 16881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c: Bootstrap starting.
I20260620 23:02:27.049701 16881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:27.050786 16881 log.cc:824] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:27.053444 16881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c: No bootstrap required, opened a new log
I20260620 23:02:27.060851 16881 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1c4a403d3c46698effde494c86dc1c" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35543 } }
I20260620 23:02:27.061174 16881 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:27.061276 16881 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1c4a403d3c46698effde494c86dc1c, State: Initialized, Role: FOLLOWER
I20260620 23:02:27.061743 16881 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1c4a403d3c46698effde494c86dc1c" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35543 } }
I20260620 23:02:27.061923 16881 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:27.062085 16881 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:27.062216 16881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:27.064085 16881 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1c4a403d3c46698effde494c86dc1c" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35543 } }
I20260620 23:02:27.064507 16881 leader_election.cc:302] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb1c4a403d3c46698effde494c86dc1c; no voters:
I20260620 23:02:27.065093 16881 leader_election.cc:288] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:27.065198 16886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:27.067099 16881 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:27.067402 16886 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [term 1 LEADER]: Becoming Leader. State: Replica: cb1c4a403d3c46698effde494c86dc1c, State: Running, Role: LEADER
I20260620 23:02:27.067869 16886 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1c4a403d3c46698effde494c86dc1c" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35543 } }
I20260620 23:02:27.070780 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1c4a403d3c46698effde494c86dc1c. Latest consensus state: current_term: 1 leader_uuid: "cb1c4a403d3c46698effde494c86dc1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1c4a403d3c46698effde494c86dc1c" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35543 } } }
I20260620 23:02:27.070756 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb1c4a403d3c46698effde494c86dc1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1c4a403d3c46698effde494c86dc1c" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 35543 } } }
I20260620 23:02:27.071272 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:27.071717 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:27.077106 16896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:27.081696 16896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:27.089920 16896 catalog_manager.cc:1347] Generated new cluster ID: afb1ee8d20fa4fc8bde3081f97fca8cd
I20260620 23:02:27.090106 16896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:27.100069 16896 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:27.101296 16896 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:27.112329 16896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb1c4a403d3c46698effde494c86dc1c: Generated new TSK 0
I20260620 23:02:27.113101 16896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:27.129670 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35543
--builtin_ntp_servers=127.10.134.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20260620 23:02:27.456455 16905 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:27.456738 16905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:27.456794 16905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:27.456871 16905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:27.463958 16905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:27.464099 16905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:27.473191 16905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35543
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:27.474399 16905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:27.476007 16905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:27.488399 16911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:27.488422 16913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:27.488418 16910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:27.489244 16905 server_base.cc:1034] running on GCE node
I20260620 23:02:27.489863 16905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:27.490798 16905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:27.492017 16905 hybrid_clock.cc:648] HybridClock initialized: now 1781996547491976 us; error 37 us; skew 500 ppm
I20260620 23:02:27.492309 16905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:27.494560 16905 webserver.cc:460] Webserver started at http://127.10.134.129:40841/ using document root <none> and password file <none>
I20260620 23:02:27.495227 16905 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:27.495352 16905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:27.495661 16905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:27.498176 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb"
format_stamp: "Formatted at 2026-06-20 23:02:27 on dist-test-slave-8k25"
I20260620 23:02:27.498979 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb"
format_stamp: "Formatted at 2026-06-20 23:02:27 on dist-test-slave-8k25"
I20260620 23:02:27.503890 16905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 23:02:27.507093 16919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:27.508301 16905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:27.508477 16905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb"
format_stamp: "Formatted at 2026-06-20 23:02:27 on dist-test-slave-8k25"
I20260620 23:02:27.508675 16905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:27.535048 16905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:27.535887 16905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:27.536186 16905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:27.537096 16905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:27.538596 16905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:27.538717 16905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:27.538851 16905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:27.538942 16905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:27.571053 16905 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:46291
I20260620 23:02:27.571105 17031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:46291 every 8 connection(s)
I20260620 23:02:27.572341 16905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:27.573017 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 16905
I20260620 23:02:27.573274 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:27.577050 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35543
--builtin_ntp_servers=127.10.134.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260620 23:02:27.586273 17032 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35543
I20260620 23:02:27.586572 17032 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:27.587240 17032 heartbeater.cc:511] Master 127.10.134.190:35543 requested a full tablet report, sending...
I20260620 23:02:27.589221 16846 ts_manager.cc:194] Registered new tserver with Master: 4ee8c4b908bd4ebda93d5ff5e3ad2aeb (127.10.134.129:46291)
I20260620 23:02:27.590746 16846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:36991
W20260620 23:02:27.893208 17036 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:27.893487 17036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:27.893595 17036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:27.893724 17036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:27.901172 17036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:27.901386 17036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:27.910288 17036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35543
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:27.911520 17036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:27.913203 17036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:27.926846 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:27.926913 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:27.927088 17036 server_base.cc:1034] running on GCE node
W20260620 23:02:27.926872 17041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:27.928009 17036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:27.928930 17036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:27.930162 17036 hybrid_clock.cc:648] HybridClock initialized: now 1781996547930100 us; error 51 us; skew 500 ppm
I20260620 23:02:27.930485 17036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:27.932821 17036 webserver.cc:460] Webserver started at http://127.10.134.130:45035/ using document root <none> and password file <none>
I20260620 23:02:27.933518 17036 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:27.933650 17036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:27.934033 17036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:27.936570 17036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b46327594d3c41abbf8a50358c72c78d"
format_stamp: "Formatted at 2026-06-20 23:02:27 on dist-test-slave-8k25"
I20260620 23:02:27.937335 17036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b46327594d3c41abbf8a50358c72c78d"
format_stamp: "Formatted at 2026-06-20 23:02:27 on dist-test-slave-8k25"
I20260620 23:02:27.942443 17036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:02:27.945700 17050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:27.947062 17036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:27.947232 17036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b46327594d3c41abbf8a50358c72c78d"
format_stamp: "Formatted at 2026-06-20 23:02:27 on dist-test-slave-8k25"
I20260620 23:02:27.947438 17036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:27.973645 17036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:27.974581 17036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:27.974885 17036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:27.975860 17036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:27.977329 17036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:27.977455 17036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:27.977581 17036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:27.977667 17036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:28.010365 17036 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:38503
I20260620 23:02:28.010418 17162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:38503 every 8 connection(s)
I20260620 23:02:28.011615 17036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:28.017846 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17036
I20260620 23:02:28.018038 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:28.021260 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35543
--builtin_ntp_servers=127.10.134.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260620 23:02:28.026476 17163 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35543
I20260620 23:02:28.026791 17163 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:28.027575 17163 heartbeater.cc:511] Master 127.10.134.190:35543 requested a full tablet report, sending...
I20260620 23:02:28.029114 16846 ts_manager.cc:194] Registered new tserver with Master: b46327594d3c41abbf8a50358c72c78d (127.10.134.130:38503)
I20260620 23:02:28.030476 16846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:37285
W20260620 23:02:28.342636 17166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:28.342923 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:28.342979 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:28.343050 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:28.350106 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:28.350246 17166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:28.359184 17166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:35543
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:28.360358 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:28.361996 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:28.374894 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:28.376461 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:28.376583 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:28.377561 17166 server_base.cc:1034] running on GCE node
I20260620 23:02:28.378224 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:28.379138 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:28.380362 17166 hybrid_clock.cc:648] HybridClock initialized: now 1781996548380287 us; error 69 us; skew 500 ppm
I20260620 23:02:28.380671 17166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:28.386695 17166 webserver.cc:460] Webserver started at http://127.10.134.131:33769/ using document root <none> and password file <none>
I20260620 23:02:28.387418 17166 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:28.387547 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:28.387873 17166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:28.390600 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d43d7c209393402d95bfc7a61a59f471"
format_stamp: "Formatted at 2026-06-20 23:02:28 on dist-test-slave-8k25"
I20260620 23:02:28.391424 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d43d7c209393402d95bfc7a61a59f471"
format_stamp: "Formatted at 2026-06-20 23:02:28 on dist-test-slave-8k25"
I20260620 23:02:28.396584 17166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:02:28.399974 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:28.401206 17166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:28.401418 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d43d7c209393402d95bfc7a61a59f471"
format_stamp: "Formatted at 2026-06-20 23:02:28 on dist-test-slave-8k25"
I20260620 23:02:28.401633 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:28.423682 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:28.424490 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:28.424798 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:28.425709 17166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:28.427172 17166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:28.427314 17166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:28.427438 17166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:28.427523 17166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:28.459998 17166 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:34825
I20260620 23:02:28.460083 17292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:34825 every 8 connection(s)
I20260620 23:02:28.461318 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:28.463555 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17166
I20260620 23:02:28.463742 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:28.475534 17293 heartbeater.cc:347] Connected to a master server at 127.10.134.190:35543
I20260620 23:02:28.475850 17293 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:28.476603 17293 heartbeater.cc:511] Master 127.10.134.190:35543 requested a full tablet report, sending...
I20260620 23:02:28.478048 16846 ts_manager.cc:194] Registered new tserver with Master: d43d7c209393402d95bfc7a61a59f471 (127.10.134.131:34825)
I20260620 23:02:28.479077 16846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:35283
I20260620 23:02:28.479800 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:28.494823 16846 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35206:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260620 23:02:28.501601 16846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:28.539800 17098 tablet_service.cc:1467] Processing CreateTablet for tablet 4e06d6ce9d5a4d298b45ce9d2b84de6e (DEFAULT_TABLE table=TestTable [id=86f12f92e619426fa0d9721a66d3365f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:28.541924 17098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e06d6ce9d5a4d298b45ce9d2b84de6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:28.548295 16967 tablet_service.cc:1467] Processing CreateTablet for tablet 4e06d6ce9d5a4d298b45ce9d2b84de6e (DEFAULT_TABLE table=TestTable [id=86f12f92e619426fa0d9721a66d3365f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:28.549818 16967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e06d6ce9d5a4d298b45ce9d2b84de6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:28.557591 17228 tablet_service.cc:1467] Processing CreateTablet for tablet 4e06d6ce9d5a4d298b45ce9d2b84de6e (DEFAULT_TABLE table=TestTable [id=86f12f92e619426fa0d9721a66d3365f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:28.558769 17228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e06d6ce9d5a4d298b45ce9d2b84de6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:28.563832 17311 tablet_bootstrap.cc:492] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb: Bootstrap starting.
I20260620 23:02:28.566161 17312 tablet_bootstrap.cc:492] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: Bootstrap starting.
I20260620 23:02:28.567836 17311 tablet_bootstrap.cc:654] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:28.569094 17311 log.cc:824] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:28.569391 17312 tablet_bootstrap.cc:654] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:28.570683 17312 log.cc:824] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:28.572427 17311 tablet_bootstrap.cc:492] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb: No bootstrap required, opened a new log
I20260620 23:02:28.572870 17311 ts_tablet_manager.cc:1397] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 23:02:28.579075 17315 tablet_bootstrap.cc:492] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471: Bootstrap starting.
I20260620 23:02:28.579782 17312 tablet_bootstrap.cc:492] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: No bootstrap required, opened a new log
I20260620 23:02:28.580049 17312 ts_tablet_manager.cc:1397] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260620 23:02:28.582902 17315 tablet_bootstrap.cc:654] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:28.584041 17315 log.cc:824] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:28.588165 17311 raft_consensus.cc:348] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.588649 17311 raft_consensus.cc:729] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee8c4b908bd4ebda93d5ff5e3ad2aeb, State: Initialized, Role: FOLLOWER
I20260620 23:02:28.589165 17311 consensus_queue.cc:260] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.590809 17032 heartbeater.cc:503] Master 127.10.134.190:35543 was elected leader, sending a full tablet report...
I20260620 23:02:28.590615 17312 raft_consensus.cc:348] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.591063 17312 raft_consensus.cc:729] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46327594d3c41abbf8a50358c72c78d, State: Initialized, Role: FOLLOWER
I20260620 23:02:28.591398 17311 ts_tablet_manager.cc:1428] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb: Time spent starting tablet: real 0.018s user 0.013s sys 0.001s
I20260620 23:02:28.591562 17312 consensus_queue.cc:260] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.592597 17163 heartbeater.cc:503] Master 127.10.134.190:35543 was elected leader, sending a full tablet report...
I20260620 23:02:28.593647 17315 tablet_bootstrap.cc:492] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471: No bootstrap required, opened a new log
I20260620 23:02:28.594031 17315 ts_tablet_manager.cc:1397] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260620 23:02:28.594262 17312 ts_tablet_manager.cc:1428] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 23:02:28.601970 17315 raft_consensus.cc:348] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.602317 17315 raft_consensus.cc:729] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43d7c209393402d95bfc7a61a59f471, State: Initialized, Role: FOLLOWER
I20260620 23:02:28.602694 17315 consensus_queue.cc:260] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.603609 17293 heartbeater.cc:503] Master 127.10.134.190:35543 was elected leader, sending a full tablet report...
I20260620 23:02:28.604028 17315 ts_tablet_manager.cc:1428] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260620 23:02:28.615602 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:28.617408 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4ee8c4b908bd4ebda93d5ff5e3ad2aeb to finish bootstrapping
I20260620 23:02:28.625921 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b46327594d3c41abbf8a50358c72c78d to finish bootstrapping
I20260620 23:02:28.633831 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d43d7c209393402d95bfc7a61a59f471 to finish bootstrapping
I20260620 23:02:28.647609 17118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e06d6ce9d5a4d298b45ce9d2b84de6e"
dest_uuid: "b46327594d3c41abbf8a50358c72c78d"
from {username='slave'} at 127.0.0.1:56890
I20260620 23:02:28.647938 17118 raft_consensus.cc:482] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:28.648147 17118 raft_consensus.cc:3037] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:28.650287 17118 raft_consensus.cc:504] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.651949 17118 leader_election.cc:288] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [CANDIDATE]: Term 1 election: Requested vote from peers d43d7c209393402d95bfc7a61a59f471 (127.10.134.131:34825), 4ee8c4b908bd4ebda93d5ff5e3ad2aeb (127.10.134.129:46291)
I20260620 23:02:28.659228 17248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e06d6ce9d5a4d298b45ce9d2b84de6e" candidate_uuid: "b46327594d3c41abbf8a50358c72c78d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d43d7c209393402d95bfc7a61a59f471"
I20260620 23:02:28.659695 17248 raft_consensus.cc:3037] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:28.659722 16987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e06d6ce9d5a4d298b45ce9d2b84de6e" candidate_uuid: "b46327594d3c41abbf8a50358c72c78d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb"
I20260620 23:02:28.660132 16987 raft_consensus.cc:3037] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:28.662261 17248 raft_consensus.cc:2454] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P d43d7c209393402d95bfc7a61a59f471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46327594d3c41abbf8a50358c72c78d in term 1.
I20260620 23:02:28.662412 16987 raft_consensus.cc:2454] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P 4ee8c4b908bd4ebda93d5ff5e3ad2aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46327594d3c41abbf8a50358c72c78d in term 1.
I20260620 23:02:28.662863 17052 leader_election.cc:302] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b46327594d3c41abbf8a50358c72c78d, d43d7c209393402d95bfc7a61a59f471; no voters:
I20260620 23:02:28.663254 17318 raft_consensus.cc:2781] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:28.663810 17318 raft_consensus.cc:686] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 LEADER]: Becoming Leader. State: Replica: b46327594d3c41abbf8a50358c72c78d, State: Running, Role: LEADER
I20260620 23:02:28.664310 17318 consensus_queue.cc:237] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:28.669322 16845 catalog_manager.cc:5495] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d reported cstate change: term changed from 0 to 1, leader changed from <none> to b46327594d3c41abbf8a50358c72c78d (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b46327594d3c41abbf8a50358c72c78d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } health_report { overall_health: UNKNOWN } } }
W20260620 23:02:28.763965 17164 tablet.cc:2369] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:29.691488 17118 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e06d6ce9d5a4d298b45ce9d2b84de6e"
dest_uuid: "b46327594d3c41abbf8a50358c72c78d"
from {username='slave'} at 127.0.0.1:56890
I20260620 23:02:29.691795 17118 raft_consensus.cc:580] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 LEADER]: Received request to step down
I20260620 23:02:29.691890 17118 raft_consensus.cc:3032] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:29.692034 17118 raft_consensus.cc:729] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b46327594d3c41abbf8a50358c72c78d, State: Running, Role: LEADER
I20260620 23:02:29.692325 17118 consensus_queue.cc:260] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b46327594d3c41abbf8a50358c72c78d" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38503 } } peers { permanent_uuid: "d43d7c209393402d95bfc7a61a59f471" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 34825 } } peers { permanent_uuid: "4ee8c4b908bd4ebda93d5ff5e3ad2aeb" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 46291 } }
I20260620 23:02:29.692651 17118 raft_consensus.cc:3037] T 4e06d6ce9d5a4d298b45ce9d2b84de6e P b46327594d3c41abbf8a50358c72c78d [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:29.694113 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16905
W20260620 23:02:29.711536 17054 connection.cc:535] client connection to 127.10.134.129:46291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:02:29.711894 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17036
W20260620 23:02:29.711844 17054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:02:29.713080 17158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:29.804143 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17036
I20260620 23:02:29.820360 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17166
I20260620 23:02:29.835892 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 16815
I20260620 23:02:29.836978 16876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:29.939934 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 16815
2026-06-20T23:02:29Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (3388 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20260620 23:02:29.979180 10778 test_util.cc:268] Using random seed: 1733966848
I20260620 23:02:29.980844 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:29.980918 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:29.980948 10778 ts_itest-base.cc:114] 5 tablet servers
I20260620 23:02:29.980994 10778 ts_itest-base.cc:115] 5 replicas per TS
I20260620 23:02:29.981029 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:29Z Disabled control of system clock
I20260620 23:02:30.011432 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:43825
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:43825
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260620 23:02:30.323770 17337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:30.324059 17337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:30.324218 17337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:30.331221 17337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:30.331315 17337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:30.331406 17337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:30.331509 17337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:30.340427 17337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:43825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:30.341735 17337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:30.343492 17337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:30.355407 17342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:30.355494 17343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:30.355460 17345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:30.356335 17337 server_base.cc:1034] running on GCE node
I20260620 23:02:30.357070 17337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:30.358582 17337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:30.359823 17337 hybrid_clock.cc:648] HybridClock initialized: now 1781996550359754 us; error 42 us; skew 500 ppm
I20260620 23:02:30.360153 17337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:30.362362 17337 webserver.cc:460] Webserver started at http://127.10.134.190:46517/ using document root <none> and password file <none>
I20260620 23:02:30.363013 17337 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:30.363152 17337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:30.363468 17337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:30.366027 17337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a567152507df48a58cfa890ef9bca5f3"
format_stamp: "Formatted at 2026-06-20 23:02:30 on dist-test-slave-8k25"
I20260620 23:02:30.366739 17337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a567152507df48a58cfa890ef9bca5f3"
format_stamp: "Formatted at 2026-06-20 23:02:30 on dist-test-slave-8k25"
I20260620 23:02:30.371735 17337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:02:30.374966 17351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:30.376225 17337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:30.376397 17337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a567152507df48a58cfa890ef9bca5f3"
format_stamp: "Formatted at 2026-06-20 23:02:30 on dist-test-slave-8k25"
I20260620 23:02:30.376590 17337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:30.390982 17337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:30.391803 17337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:30.392123 17337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:30.409747 17337 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:43825
I20260620 23:02:30.409765 17402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:43825 every 8 connection(s)
I20260620 23:02:30.411279 17337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:30.414921 17403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:30.419459 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17337
I20260620 23:02:30.419685 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:30.425522 17403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3: Bootstrap starting.
I20260620 23:02:30.429512 17403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:30.430564 17403 log.cc:824] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:30.433508 17403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3: No bootstrap required, opened a new log
I20260620 23:02:30.441605 17403 raft_consensus.cc:348] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a567152507df48a58cfa890ef9bca5f3" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43825 } }
I20260620 23:02:30.441988 17403 raft_consensus.cc:374] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:30.442091 17403 raft_consensus.cc:729] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a567152507df48a58cfa890ef9bca5f3, State: Initialized, Role: FOLLOWER
I20260620 23:02:30.442735 17403 consensus_queue.cc:260] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a567152507df48a58cfa890ef9bca5f3" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43825 } }
I20260620 23:02:30.443020 17403 raft_consensus.cc:388] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:30.443169 17403 raft_consensus.cc:482] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:30.443333 17403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:30.445233 17403 raft_consensus.cc:504] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a567152507df48a58cfa890ef9bca5f3" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43825 } }
I20260620 23:02:30.445690 17403 leader_election.cc:302] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a567152507df48a58cfa890ef9bca5f3; no voters:
I20260620 23:02:30.446273 17403 leader_election.cc:288] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:30.446399 17408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:30.447088 17408 raft_consensus.cc:686] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [term 1 LEADER]: Becoming Leader. State: Replica: a567152507df48a58cfa890ef9bca5f3, State: Running, Role: LEADER
I20260620 23:02:30.447784 17408 consensus_queue.cc:237] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a567152507df48a58cfa890ef9bca5f3" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43825 } }
I20260620 23:02:30.448397 17403 sys_catalog.cc:564] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:30.450747 17410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a567152507df48a58cfa890ef9bca5f3. Latest consensus state: current_term: 1 leader_uuid: "a567152507df48a58cfa890ef9bca5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a567152507df48a58cfa890ef9bca5f3" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43825 } } }
I20260620 23:02:30.450719 17409 sys_catalog.cc:455] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a567152507df48a58cfa890ef9bca5f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a567152507df48a58cfa890ef9bca5f3" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 43825 } } }
I20260620 23:02:30.451167 17410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:30.451215 17409 sys_catalog.cc:458] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:30.452962 17417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:30.458156 17417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:30.467043 17417 catalog_manager.cc:1347] Generated new cluster ID: fd374c0cad694e708d10d4a0498ffa68
I20260620 23:02:30.467159 17417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:30.509543 17417 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:30.511377 17417 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:30.520190 17417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a567152507df48a58cfa890ef9bca5f3: Generated new TSK 0
I20260620 23:02:30.521186 17417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:30.531378 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--builtin_ntp_servers=127.10.134.148:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260620 23:02:30.856705 17427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:30.857002 17427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:30.857120 17427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:30.857246 17427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:30.864532 17427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:30.864717 17427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:30.873781 17427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:30.875244 17427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:30.876948 17427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:30.890110 17435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:30.890173 17433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:30.890151 17432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:30.891052 17427 server_base.cc:1034] running on GCE node
I20260620 23:02:30.891610 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:30.892589 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:30.893822 17427 hybrid_clock.cc:648] HybridClock initialized: now 1781996550893778 us; error 30 us; skew 500 ppm
I20260620 23:02:30.894229 17427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:30.896780 17427 webserver.cc:460] Webserver started at http://127.10.134.129:37681/ using document root <none> and password file <none>
I20260620 23:02:30.897473 17427 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:30.897585 17427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:30.897895 17427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:30.900449 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3add3eb52cad4cb7a54c40ec26d6cab4"
format_stamp: "Formatted at 2026-06-20 23:02:30 on dist-test-slave-8k25"
I20260620 23:02:30.901165 17427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3add3eb52cad4cb7a54c40ec26d6cab4"
format_stamp: "Formatted at 2026-06-20 23:02:30 on dist-test-slave-8k25"
I20260620 23:02:30.906190 17427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:02:30.909547 17441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:30.911042 17427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:30.911223 17427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3add3eb52cad4cb7a54c40ec26d6cab4"
format_stamp: "Formatted at 2026-06-20 23:02:30 on dist-test-slave-8k25"
I20260620 23:02:30.911451 17427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:30.935307 17427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:30.936151 17427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:30.936458 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:30.937391 17427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:30.938818 17427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:30.938936 17427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:30.939059 17427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:30.939147 17427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:30.971475 17427 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:42745
I20260620 23:02:30.971560 17553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:42745 every 8 connection(s)
I20260620 23:02:30.972721 17427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:30.978293 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17427
I20260620 23:02:30.978477 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:30.981868 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--builtin_ntp_servers=127.10.134.148:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:30.987166 17554 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43825
I20260620 23:02:30.987476 17554 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:30.988183 17554 heartbeater.cc:511] Master 127.10.134.190:43825 requested a full tablet report, sending...
I20260620 23:02:30.990207 17368 ts_manager.cc:194] Registered new tserver with Master: 3add3eb52cad4cb7a54c40ec26d6cab4 (127.10.134.129:42745)
I20260620 23:02:30.991657 17368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:53043
W20260620 23:02:31.297261 17558 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:31.297600 17558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:31.297715 17558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:31.297839 17558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:31.305136 17558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:31.305346 17558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:31.315023 17558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:31.316270 17558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:31.318001 17558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:31.330677 17563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:31.330705 17564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:31.330643 17566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:31.331456 17558 server_base.cc:1034] running on GCE node
I20260620 23:02:31.332086 17558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:31.332957 17558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:31.334175 17558 hybrid_clock.cc:648] HybridClock initialized: now 1781996551334128 us; error 43 us; skew 500 ppm
I20260620 23:02:31.334496 17558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:31.336627 17558 webserver.cc:460] Webserver started at http://127.10.134.130:40339/ using document root <none> and password file <none>
I20260620 23:02:31.337289 17558 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:31.337417 17558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:31.337723 17558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:31.340222 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "40cb509037e94527974612fa626fb868"
format_stamp: "Formatted at 2026-06-20 23:02:31 on dist-test-slave-8k25"
I20260620 23:02:31.340950 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "40cb509037e94527974612fa626fb868"
format_stamp: "Formatted at 2026-06-20 23:02:31 on dist-test-slave-8k25"
I20260620 23:02:31.345876 17558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:02:31.349248 17572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:31.350742 17558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:31.350903 17558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "40cb509037e94527974612fa626fb868"
format_stamp: "Formatted at 2026-06-20 23:02:31 on dist-test-slave-8k25"
I20260620 23:02:31.351112 17558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:31.368495 17558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:31.369306 17558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:31.369607 17558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:31.370633 17558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:31.372027 17558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:31.372174 17558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:31.372308 17558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:31.372393 17558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:31.405239 17558 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:42455
I20260620 23:02:31.405292 17684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:42455 every 8 connection(s)
I20260620 23:02:31.406515 17558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:31.413210 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17558
I20260620 23:02:31.413389 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:31.416790 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--builtin_ntp_servers=127.10.134.148:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:31.421510 17685 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43825
I20260620 23:02:31.421824 17685 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:31.422554 17685 heartbeater.cc:511] Master 127.10.134.190:43825 requested a full tablet report, sending...
I20260620 23:02:31.423827 17368 ts_manager.cc:194] Registered new tserver with Master: 40cb509037e94527974612fa626fb868 (127.10.134.130:42455)
I20260620 23:02:31.425073 17368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:36283
W20260620 23:02:31.739887 17688 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:31.740171 17688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:31.740233 17688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:31.740303 17688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:31.747457 17688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:31.747597 17688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:31.756882 17688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:31.758183 17688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:31.759864 17688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:31.772469 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:31.772524 17696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:31.772555 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:31.773358 17688 server_base.cc:1034] running on GCE node
I20260620 23:02:31.773909 17688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:31.774850 17688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:31.776052 17688 hybrid_clock.cc:648] HybridClock initialized: now 1781996551776016 us; error 30 us; skew 500 ppm
I20260620 23:02:31.776362 17688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:31.778568 17688 webserver.cc:460] Webserver started at http://127.10.134.131:43851/ using document root <none> and password file <none>
I20260620 23:02:31.779337 17688 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:31.779428 17688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:31.779764 17688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:31.782473 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "341557708fdd4461aea870367dde9b68"
format_stamp: "Formatted at 2026-06-20 23:02:31 on dist-test-slave-8k25"
I20260620 23:02:31.783221 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "341557708fdd4461aea870367dde9b68"
format_stamp: "Formatted at 2026-06-20 23:02:31 on dist-test-slave-8k25"
I20260620 23:02:31.788019 17688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:02:31.791296 17702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:31.792544 17688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:31.792704 17688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "341557708fdd4461aea870367dde9b68"
format_stamp: "Formatted at 2026-06-20 23:02:31 on dist-test-slave-8k25"
I20260620 23:02:31.792910 17688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:31.812600 17688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:31.813400 17688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:31.813709 17688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:31.814688 17688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:31.816092 17688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:31.816200 17688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:31.816322 17688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:31.816398 17688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:31.849614 17688 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:46607
I20260620 23:02:31.849722 17814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:46607 every 8 connection(s)
I20260620 23:02:31.850950 17688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:31.852006 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17688
I20260620 23:02:31.852169 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:31.855687 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.132:0
--local_ip_for_outbound_sockets=127.10.134.132
--webserver_interface=127.10.134.132
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--builtin_ntp_servers=127.10.134.148:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:31.865815 17815 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43825
I20260620 23:02:31.866150 17815 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:31.866899 17815 heartbeater.cc:511] Master 127.10.134.190:43825 requested a full tablet report, sending...
I20260620 23:02:31.868232 17368 ts_manager.cc:194] Registered new tserver with Master: 341557708fdd4461aea870367dde9b68 (127.10.134.131:46607)
I20260620 23:02:31.868995 17368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:55031
I20260620 23:02:31.994472 17554 heartbeater.cc:503] Master 127.10.134.190:43825 was elected leader, sending a full tablet report...
W20260620 23:02:32.178517 17818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:32.178866 17818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:32.178977 17818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:32.179103 17818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:32.186460 17818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:32.186666 17818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.132
I20260620 23:02:32.195680 17818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.10.134.132
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.132
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:32.197031 17818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:32.198755 17818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:32.214272 17823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:32.214244 17824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:32.214200 17826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:32.215027 17818 server_base.cc:1034] running on GCE node
I20260620 23:02:32.215647 17818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:32.216640 17818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:32.217852 17818 hybrid_clock.cc:648] HybridClock initialized: now 1781996552217806 us; error 43 us; skew 500 ppm
I20260620 23:02:32.218189 17818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:32.220393 17818 webserver.cc:460] Webserver started at http://127.10.134.132:36005/ using document root <none> and password file <none>
I20260620 23:02:32.221078 17818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:32.221203 17818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:32.221518 17818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:32.224045 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa"
format_stamp: "Formatted at 2026-06-20 23:02:32 on dist-test-slave-8k25"
I20260620 23:02:32.224850 17818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa"
format_stamp: "Formatted at 2026-06-20 23:02:32 on dist-test-slave-8k25"
I20260620 23:02:32.229723 17818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 23:02:32.233040 17832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:32.234406 17818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:32.234586 17818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa"
format_stamp: "Formatted at 2026-06-20 23:02:32 on dist-test-slave-8k25"
I20260620 23:02:32.234802 17818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:32.257380 17818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:32.258292 17818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:32.258602 17818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:32.259552 17818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:32.261183 17818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:32.261327 17818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:32.261487 17818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:32.261583 17818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:32.295811 17818 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.132:42687
I20260620 23:02:32.295928 17944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.132:42687 every 8 connection(s)
I20260620 23:02:32.297322 17818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260620 23:02:32.307559 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17818
I20260620 23:02:32.307797 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260620 23:02:32.311180 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.133:0
--local_ip_for_outbound_sockets=127.10.134.133
--webserver_interface=127.10.134.133
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--builtin_ntp_servers=127.10.134.148:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:32.311365 17945 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43825
I20260620 23:02:32.311699 17945 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:32.312377 17945 heartbeater.cc:511] Master 127.10.134.190:43825 requested a full tablet report, sending...
I20260620 23:02:32.313752 17368 ts_manager.cc:194] Registered new tserver with Master: 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132:42687)
I20260620 23:02:32.314551 17368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.132:39853
I20260620 23:02:32.427429 17685 heartbeater.cc:503] Master 127.10.134.190:43825 was elected leader, sending a full tablet report...
W20260620 23:02:32.651014 17948 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:32.651489 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:32.651631 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:32.651763 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:32.659338 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:32.659546 17948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.133
I20260620 23:02:32.668613 17948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.10.134.133
--webserver_port=0
--tserver_master_addrs=127.10.134.190:43825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.133
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:32.669890 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:32.671638 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:32.684878 17953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:32.684924 17954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:32.684844 17956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:32.685779 17948 server_base.cc:1034] running on GCE node
I20260620 23:02:32.686394 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:32.687309 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:32.688539 17948 hybrid_clock.cc:648] HybridClock initialized: now 1781996552688483 us; error 50 us; skew 500 ppm
I20260620 23:02:32.688854 17948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:32.691166 17948 webserver.cc:460] Webserver started at http://127.10.134.133:38861/ using document root <none> and password file <none>
I20260620 23:02:32.691851 17948 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:32.691996 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:32.692298 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:32.694824 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f38061dc6adc44e2a13027cd95d9d08c"
format_stamp: "Formatted at 2026-06-20 23:02:32 on dist-test-slave-8k25"
I20260620 23:02:32.695581 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f38061dc6adc44e2a13027cd95d9d08c"
format_stamp: "Formatted at 2026-06-20 23:02:32 on dist-test-slave-8k25"
I20260620 23:02:32.700687 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:02:32.703950 17962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:32.705235 17948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:32.705422 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f38061dc6adc44e2a13027cd95d9d08c"
format_stamp: "Formatted at 2026-06-20 23:02:32 on dist-test-slave-8k25"
I20260620 23:02:32.705632 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:32.742430 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:32.743275 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:32.743573 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:32.744510 17948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:32.745898 17948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:32.746045 17948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:32.746171 17948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:32.746254 17948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:32.779409 17948 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.133:39635
I20260620 23:02:32.779461 18074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.133:39635 every 8 connection(s)
I20260620 23:02:32.780925 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260620 23:02:32.782178 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 17948
I20260620 23:02:32.782327 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260620 23:02:32.794348 18075 heartbeater.cc:347] Connected to a master server at 127.10.134.190:43825
I20260620 23:02:32.794643 18075 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:32.795344 18075 heartbeater.cc:511] Master 127.10.134.190:43825 requested a full tablet report, sending...
I20260620 23:02:32.796782 17368 ts_manager.cc:194] Registered new tserver with Master: f38061dc6adc44e2a13027cd95d9d08c (127.10.134.133:39635)
I20260620 23:02:32.797835 17368 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.133:45323
I20260620 23:02:32.798363 10778 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260620 23:02:32.814813 17368 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35726:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260620 23:02:32.821811 17368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260620 23:02:32.861245 17750 tablet_service.cc:1467] Processing CreateTablet for tablet 965a62341cd543f3a8136a59fe7db0d6 (DEFAULT_TABLE table=TestTable [id=0eac624d7045481097fb5222a0ba85a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:32.863016 17750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965a62341cd543f3a8136a59fe7db0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:32.866523 18010 tablet_service.cc:1467] Processing CreateTablet for tablet 965a62341cd543f3a8136a59fe7db0d6 (DEFAULT_TABLE table=TestTable [id=0eac624d7045481097fb5222a0ba85a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:32.867892 18010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965a62341cd543f3a8136a59fe7db0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:32.867969 17620 tablet_service.cc:1467] Processing CreateTablet for tablet 965a62341cd543f3a8136a59fe7db0d6 (DEFAULT_TABLE table=TestTable [id=0eac624d7045481097fb5222a0ba85a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:32.871389 17815 heartbeater.cc:503] Master 127.10.134.190:43825 was elected leader, sending a full tablet report...
I20260620 23:02:32.871506 17489 tablet_service.cc:1467] Processing CreateTablet for tablet 965a62341cd543f3a8136a59fe7db0d6 (DEFAULT_TABLE table=TestTable [id=0eac624d7045481097fb5222a0ba85a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:32.872740 17489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965a62341cd543f3a8136a59fe7db0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:32.874346 17620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965a62341cd543f3a8136a59fe7db0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:32.875510 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 965a62341cd543f3a8136a59fe7db0d6 (DEFAULT_TABLE table=TestTable [id=0eac624d7045481097fb5222a0ba85a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:32.877084 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965a62341cd543f3a8136a59fe7db0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:32.906213 18096 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: Bootstrap starting.
I20260620 23:02:32.907312 18097 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: Bootstrap starting.
I20260620 23:02:32.908569 18099 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: Bootstrap starting.
I20260620 23:02:32.911218 18097 tablet_bootstrap.cc:654] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:32.911218 18096 tablet_bootstrap.cc:654] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:32.912504 18096 log.cc:824] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:32.912504 18097 log.cc:824] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:32.912652 18099 tablet_bootstrap.cc:654] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:32.915655 18100 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: Bootstrap starting.
I20260620 23:02:32.919277 18096 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: No bootstrap required, opened a new log
I20260620 23:02:32.919708 18096 ts_tablet_manager.cc:1397] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260620 23:02:32.920053 18100 tablet_bootstrap.cc:654] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:32.921543 18100 log.cc:824] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:32.923597 18098 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: Bootstrap starting.
I20260620 23:02:32.924221 18099 log.cc:824] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:32.926858 18098 tablet_bootstrap.cc:654] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:32.927846 18098 log.cc:824] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:32.930439 18098 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: No bootstrap required, opened a new log
I20260620 23:02:32.930841 18098 ts_tablet_manager.cc:1397] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 23:02:32.932114 18097 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: No bootstrap required, opened a new log
I20260620 23:02:32.932509 18097 ts_tablet_manager.cc:1397] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.012s
I20260620 23:02:32.944756 18098 raft_consensus.cc:348] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.945581 18098 raft_consensus.cc:729] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a25f54fcd64dfb9d79a6dc16bbddfa, State: Initialized, Role: FOLLOWER
I20260620 23:02:32.946506 18098 consensus_queue.cc:260] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.950780 18096 raft_consensus.cc:348] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.952371 18097 raft_consensus.cc:348] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.953076 18096 raft_consensus.cc:729] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341557708fdd4461aea870367dde9b68, State: Initialized, Role: FOLLOWER
I20260620 23:02:32.953326 18098 ts_tablet_manager.cc:1428] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260620 23:02:32.953640 18096 consensus_queue.cc:260] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.953948 17945 heartbeater.cc:503] Master 127.10.134.190:43825 was elected leader, sending a full tablet report...
I20260620 23:02:32.953925 18097 raft_consensus.cc:729] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38061dc6adc44e2a13027cd95d9d08c, State: Initialized, Role: FOLLOWER
I20260620 23:02:32.954422 18097 consensus_queue.cc:260] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.954762 18099 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: No bootstrap required, opened a new log
I20260620 23:02:32.955749 18100 tablet_bootstrap.cc:492] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: No bootstrap required, opened a new log
I20260620 23:02:32.956183 18100 ts_tablet_manager.cc:1397] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.024s
I20260620 23:02:32.962358 18096 ts_tablet_manager.cc:1428] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: Time spent starting tablet: real 0.042s user 0.018s sys 0.005s
I20260620 23:02:32.964128 18099 ts_tablet_manager.cc:1397] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.023s
I20260620 23:02:32.967484 18075 heartbeater.cc:503] Master 127.10.134.190:43825 was elected leader, sending a full tablet report...
I20260620 23:02:32.968161 18097 ts_tablet_manager.cc:1428] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: Time spent starting tablet: real 0.035s user 0.027s sys 0.000s
I20260620 23:02:32.970741 18100 raft_consensus.cc:348] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.971400 18100 raft_consensus.cc:729] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3add3eb52cad4cb7a54c40ec26d6cab4, State: Initialized, Role: FOLLOWER
I20260620 23:02:32.971925 18100 consensus_queue.cc:260] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.973670 18100 ts_tablet_manager.cc:1428] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260620 23:02:32.974192 18099 raft_consensus.cc:348] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:32.974680 18099 raft_consensus.cc:729] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cb509037e94527974612fa626fb868, State: Initialized, Role: FOLLOWER
I20260620 23:02:32.975147 18099 consensus_queue.cc:260] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
W20260620 23:02:32.975553 17555 tablet.cc:2369] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:32.977623 18099 ts_tablet_manager.cc:1428] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260620 23:02:32.990164 10778 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260620 23:02:32.992308 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3add3eb52cad4cb7a54c40ec26d6cab4 to finish bootstrapping
I20260620 23:02:33.000979 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 40cb509037e94527974612fa626fb868 to finish bootstrapping
I20260620 23:02:33.008706 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 341557708fdd4461aea870367dde9b68 to finish bootstrapping
I20260620 23:02:33.016069 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 99a25f54fcd64dfb9d79a6dc16bbddfa to finish bootstrapping
I20260620 23:02:33.023049 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f38061dc6adc44e2a13027cd95d9d08c to finish bootstrapping
W20260620 23:02:33.032752 18076 tablet.cc:2369] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:33.036723 17640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6"
dest_uuid: "40cb509037e94527974612fa626fb868"
from {username='slave'} at 127.0.0.1:33844
I20260620 23:02:33.037009 17640 raft_consensus.cc:482] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:33.037168 17640 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:33.039462 17640 raft_consensus.cc:504] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:33.041766 17640 leader_election.cc:288] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [CANDIDATE]: Term 1 election: Requested vote from peers 341557708fdd4461aea870367dde9b68 (127.10.134.131:46607), f38061dc6adc44e2a13027cd95d9d08c (127.10.134.133:39635), 3add3eb52cad4cb7a54c40ec26d6cab4 (127.10.134.129:42745), 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132:42687)
W20260620 23:02:33.049433 17946 tablet.cc:2369] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:33.055998 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "40cb509037e94527974612fa626fb868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "341557708fdd4461aea870367dde9b68"
I20260620 23:02:33.056429 17770 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:33.056995 18030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "40cb509037e94527974612fa626fb868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f38061dc6adc44e2a13027cd95d9d08c"
I20260620 23:02:33.057387 18030 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:33.059700 17770 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cb509037e94527974612fa626fb868 in term 1.
I20260620 23:02:33.059692 17900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "40cb509037e94527974612fa626fb868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa"
I20260620 23:02:33.060091 17900 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:33.060472 18030 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cb509037e94527974612fa626fb868 in term 1.
I20260620 23:02:33.060997 17509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "40cb509037e94527974612fa626fb868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4"
I20260620 23:02:33.061342 17509 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:33.061733 17573 leader_election.cc:302] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341557708fdd4461aea870367dde9b68, 40cb509037e94527974612fa626fb868, f38061dc6adc44e2a13027cd95d9d08c; no voters:
I20260620 23:02:33.062227 18110 raft_consensus.cc:2781] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:33.062816 18110 raft_consensus.cc:686] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 LEADER]: Becoming Leader. State: Replica: 40cb509037e94527974612fa626fb868, State: Running, Role: LEADER
I20260620 23:02:33.062858 17900 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cb509037e94527974612fa626fb868 in term 1.
I20260620 23:02:33.063421 18110 consensus_queue.cc:237] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }
I20260620 23:02:33.064431 17509 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40cb509037e94527974612fa626fb868 in term 1.
I20260620 23:02:33.068274 17368 catalog_manager.cc:5495] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40cb509037e94527974612fa626fb868 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "40cb509037e94527974612fa626fb868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } health_report { overall_health: UNKNOWN } } }
W20260620 23:02:33.103256 17816 tablet.cc:2369] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:02:33.159093 17686 tablet.cc:2369] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:33.468276 18110 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:33.526157 18110 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:33.534047 18124 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:33.534839 18124 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:33.536156 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17337
I20260620 23:02:33.537427 17398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:33.653872 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17337
W20260620 23:02:33.674104 17574 connection.cc:535] client connection to 127.10.134.190:43825 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:02:33.674625 10778 raft_consensus_election-itest.cc:666] Removing tserver with uuid 99a25f54fcd64dfb9d79a6dc16bbddfa
W20260620 23:02:33.674801 17685 heartbeater.cc:644] Failed to heartbeat to 127.10.134.190:43825 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 23:02:33.675035 17945 heartbeater.cc:644] Failed to heartbeat to 127.10.134.190:43825 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.10.134.190:43825 (error 108)
W20260620 23:02:33.675166 18075 heartbeater.cc:644] Failed to heartbeat to 127.10.134.190:43825 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.10.134.190:43825 (error 108)
I20260620 23:02:33.677292 17640 consensus_queue.cc:237] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } }
I20260620 23:02:33.679246 18030 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:33.679293 17509 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:33.679497 17770 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:33.679926 18117 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 23:02:33.680539 18124 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 23:02:33.680934 18110 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 23:02:33.683238 18124 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } }
I20260620 23:02:33.683889 17509 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } }
I20260620 23:02:33.684207 17770 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } }
I20260620 23:02:33.688234 18030 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } }
W20260620 23:02:33.692363 17554 heartbeater.cc:644] Failed to heartbeat to 127.10.134.190:43825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.134.190:43825: connect: Connection refused (error 111)
W20260620 23:02:33.695497 17815 heartbeater.cc:644] Failed to heartbeat to 127.10.134.190:43825 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.134.190:43825: connect: Connection refused (error 111)
I20260620 23:02:33.715402 10778 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20260620 23:02:33.748109 10778 raft_consensus_election-itest.cc:684] Adding back Peer 99a25f54fcd64dfb9d79a6dc16bbddfa and expecting timeout...
I20260620 23:02:33.749771 17640 consensus_queue.cc:237] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } }
I20260620 23:02:33.751889 17900 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 23:02:33.752485 18142 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 23:02:33.755003 17900 raft_consensus.cc:858] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } }
W20260620 23:02:33.755146 17900 raft_consensus.cc:1565] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } }
I20260620 23:02:33.755725 17900 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } }
I20260620 23:02:33.759755 17900 raft_consensus.cc:1467] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260620 23:02:33.849562 10778 raft_consensus_election-itest.cc:688] Timeout achieved.
I20260620 23:02:33.858137 10778 raft_consensus_election-itest.cc:699] Forcing Peer 40cb509037e94527974612fa626fb868 to step down...
I20260620 23:02:33.858793 17640 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6"
dest_uuid: "40cb509037e94527974612fa626fb868"
from {username='slave'} at 127.0.0.1:33844
I20260620 23:02:33.859068 17640 raft_consensus.cc:580] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 LEADER]: Received request to step down
I20260620 23:02:33.859166 17640 raft_consensus.cc:3032] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:33.859275 17640 raft_consensus.cc:729] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 40cb509037e94527974612fa626fb868, State: Running, Role: LEADER
I20260620 23:02:33.859549 17640 consensus_queue.cc:260] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } }
I20260620 23:02:33.859934 17640 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:33.860500 10778 raft_consensus_election-itest.cc:705] Resuming Peer 3add3eb52cad4cb7a54c40ec26d6cab4 ...
I20260620 23:02:33.861233 10778 raft_consensus_election-itest.cc:710] Trying to elect Peer 99a25f54fcd64dfb9d79a6dc16bbddfa ...
I20260620 23:02:33.861663 17509 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 23:02:33.862003 17900 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6"
dest_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa"
from {username='slave'} at 127.0.0.1:39272
I20260620 23:02:33.862314 17900 raft_consensus.cc:482] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:33.862454 17900 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:33.864355 17900 raft_consensus.cc:504] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } }
I20260620 23:02:33.865967 17900 leader_election.cc:288] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [CANDIDATE]: Term 2 election: Requested vote from peers 341557708fdd4461aea870367dde9b68 (127.10.134.131:46607), f38061dc6adc44e2a13027cd95d9d08c (127.10.134.133:39635), 40cb509037e94527974612fa626fb868 (127.10.134.130:42455), 3add3eb52cad4cb7a54c40ec26d6cab4 (127.10.134.129:42745)
I20260620 23:02:33.881182 10778 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20260620 23:02:33.884805 17509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4"
I20260620 23:02:33.884836 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "40cb509037e94527974612fa626fb868"
I20260620 23:02:33.885031 17509 raft_consensus.cc:1799] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 1 FOLLOWER]: Handling vote request from an unknown peer 99a25f54fcd64dfb9d79a6dc16bbddfa
I20260620 23:02:33.885169 17509 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:33.887024 17509 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a25f54fcd64dfb9d79a6dc16bbddfa in term 2.
I20260620 23:02:33.887032 17640 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a25f54fcd64dfb9d79a6dc16bbddfa in term 2.
I20260620 23:02:33.887542 17835 leader_election.cc:302] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 3add3eb52cad4cb7a54c40ec26d6cab4, 40cb509037e94527974612fa626fb868, 99a25f54fcd64dfb9d79a6dc16bbddfa; no voters:
I20260620 23:02:33.887969 18126 raft_consensus.cc:2781] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:02:33.888501 18126 raft_consensus.cc:686] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 2 LEADER]: Becoming Leader. State: Replica: 99a25f54fcd64dfb9d79a6dc16bbddfa, State: Running, Role: LEADER
I20260620 23:02:33.889030 18126 consensus_queue.cc:237] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } }
I20260620 23:02:33.984437 10778 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260620 23:02:34.187963 10778 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260620 23:02:34.297621 17640 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 2 FOLLOWER]: Refusing update from remote peer 99a25f54fcd64dfb9d79a6dc16bbddfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 23:02:34.298501 18126 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 23:02:34.310320 17509 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 2 FOLLOWER]: Refusing update from remote peer 99a25f54fcd64dfb9d79a6dc16bbddfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 23:02:34.310855 18126 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 23:02:34.313687 18148 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } } }
I20260620 23:02:34.314221 17509 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 3add3eb52cad4cb7a54c40ec26d6cab4 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } } }
I20260620 23:02:34.319837 17640 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 40cb509037e94527974612fa626fb868 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } } }
W20260620 23:02:34.324811 17640 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33844 (request call id 48) took 575 ms (client timeout 100 ms). Trace:
W20260620 23:02:34.325022 17640 rpcz_store.cc:269] 0620 23:02:33.748955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:33.749042 (+ 87us) service_pool.cc:227] Handling call
0620 23:02:33.750772 (+ 1730us) log.cc:842] Serialized 618 byte log entry
0620 23:02:34.324746 (+573974us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":867,"raft.run_cpu_time_us":624,"raft.run_wall_time_us":625,"spinlock_wait_cycles":19200,"thread_start_us":555,"threads_started":1}
I20260620 23:02:34.491570 10778 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20260620 23:02:34.493808 17770 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 23:02:34.494742 18030 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 1 FOLLOWER]: Refusing update from remote peer 40cb509037e94527974612fa626fb868: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260620 23:02:34.506446 17770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "341557708fdd4461aea870367dde9b68"
I20260620 23:02:34.506669 17770 raft_consensus.cc:1799] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 1 FOLLOWER]: Handling vote request from an unknown peer 99a25f54fcd64dfb9d79a6dc16bbddfa
I20260620 23:02:34.506784 17770 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:34.509685 17770 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a25f54fcd64dfb9d79a6dc16bbddfa in term 2.
I20260620 23:02:34.510507 17769 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 2 FOLLOWER]: Refusing update from remote peer 99a25f54fcd64dfb9d79a6dc16bbddfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 23:02:34.511193 18155 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 23:02:34.518026 18030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965a62341cd543f3a8136a59fe7db0d6" candidate_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f38061dc6adc44e2a13027cd95d9d08c"
I20260620 23:02:34.518230 18030 raft_consensus.cc:1799] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 1 FOLLOWER]: Handling vote request from an unknown peer 99a25f54fcd64dfb9d79a6dc16bbddfa
I20260620 23:02:34.518349 18030 raft_consensus.cc:3037] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:34.521063 18030 raft_consensus.cc:2454] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99a25f54fcd64dfb9d79a6dc16bbddfa in term 2.
I20260620 23:02:34.521387 18029 raft_consensus.cc:1261] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 2 FOLLOWER]: Refusing update from remote peer 99a25f54fcd64dfb9d79a6dc16bbddfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260620 23:02:34.522022 18155 consensus_queue.cc:1035] T 965a62341cd543f3a8136a59fe7db0d6 P 99a25f54fcd64dfb9d79a6dc16bbddfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 23:02:34.537444 18030 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P f38061dc6adc44e2a13027cd95d9d08c [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } } }
I20260620 23:02:34.542336 17769 raft_consensus.cc:2932] T 965a62341cd543f3a8136a59fe7db0d6 P 341557708fdd4461aea870367dde9b68 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 99a25f54fcd64dfb9d79a6dc16bbddfa (127.10.134.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "341557708fdd4461aea870367dde9b68" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 46607 } } peers { permanent_uuid: "f38061dc6adc44e2a13027cd95d9d08c" member_type: VOTER last_known_addr { host: "127.10.134.133" port: 39635 } } peers { permanent_uuid: "40cb509037e94527974612fa626fb868" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42455 } } peers { permanent_uuid: "3add3eb52cad4cb7a54c40ec26d6cab4" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42745 } } peers { permanent_uuid: "99a25f54fcd64dfb9d79a6dc16bbddfa" member_type: VOTER last_known_addr { host: "127.10.134.132" port: 42687 } attrs { } } }
I20260620 23:02:34.545375 18170 mvcc.cc:205] Tried to move back new op lower bound from 7299057887226974208 to 7299057884735815680. Current Snapshot: MvccSnapshot[applied={T|T < 7299057887226974208}]
I20260620 23:02:34.550549 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17427
I20260620 23:02:34.551666 17549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:34.639389 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17427
I20260620 23:02:34.657042 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17558
I20260620 23:02:34.658126 17680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:34.751060 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17558
I20260620 23:02:34.768173 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17688
I20260620 23:02:34.769308 17810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:34.856900 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17688
I20260620 23:02:34.874914 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17818
I20260620 23:02:34.875929 17940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:34.967813 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17818
I20260620 23:02:34.984212 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 17948
I20260620 23:02:34.985261 18070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:35.072062 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 17948
2026-06-20T23:02:35Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (5131 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20260620 23:02:35.110334 10778 test_util.cc:268] Using random seed: 1739098009
I20260620 23:02:35.112085 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:35.112156 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:35.112228 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:02:35.112286 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:02:35.112362 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:35Z Disabled control of system clock
I20260620 23:02:35.141461 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:33929
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:33929
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260620 23:02:35.466856 18186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:35.467144 18186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:35.467290 18186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:35.474684 18186 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:35.474818 18186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:35.474907 18186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:35.475003 18186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:35.475088 18186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:35.484297 18186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:33929
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:33929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:35.485580 18186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:35.487349 18186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:35.498661 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:35.498745 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:35.498787 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:35.499691 18186 server_base.cc:1034] running on GCE node
I20260620 23:02:35.500399 18186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:35.501744 18186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:35.502966 18186 hybrid_clock.cc:648] HybridClock initialized: now 1781996555502928 us; error 58 us; skew 500 ppm
I20260620 23:02:35.503310 18186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:35.505430 18186 webserver.cc:460] Webserver started at http://127.10.134.190:46603/ using document root <none> and password file <none>
I20260620 23:02:35.506163 18186 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:35.506287 18186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:35.506644 18186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:35.509189 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8114cfbce722414a981bdaa62faa4b4a"
format_stamp: "Formatted at 2026-06-20 23:02:35 on dist-test-slave-8k25"
I20260620 23:02:35.509919 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8114cfbce722414a981bdaa62faa4b4a"
format_stamp: "Formatted at 2026-06-20 23:02:35 on dist-test-slave-8k25"
I20260620 23:02:35.514842 18186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:02:35.518018 18200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:35.519255 18186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 23:02:35.519421 18186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8114cfbce722414a981bdaa62faa4b4a"
format_stamp: "Formatted at 2026-06-20 23:02:35 on dist-test-slave-8k25"
I20260620 23:02:35.519630 18186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:35.541126 18186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:35.541980 18186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:35.542264 18186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:35.560313 18186 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:33929
I20260620 23:02:35.560320 18251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:33929 every 8 connection(s)
I20260620 23:02:35.561661 18186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:35.565289 18252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:35.569454 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18186
I20260620 23:02:35.569670 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:35.576063 18252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Bootstrap starting.
I20260620 23:02:35.579516 18252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:35.580672 18252 log.cc:824] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:35.583465 18252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: No bootstrap required, opened a new log
I20260620 23:02:35.590291 18252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:35.590653 18252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:35.590755 18252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8114cfbce722414a981bdaa62faa4b4a, State: Initialized, Role: FOLLOWER
I20260620 23:02:35.591256 18252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:35.591485 18252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:35.591629 18252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:35.591789 18252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:35.593717 18252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:35.594215 18252 leader_election.cc:302] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8114cfbce722414a981bdaa62faa4b4a; no voters:
I20260620 23:02:35.594738 18252 leader_election.cc:288] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:35.594861 18257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:35.596240 18257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 LEADER]: Becoming Leader. State: Replica: 8114cfbce722414a981bdaa62faa4b4a, State: Running, Role: LEADER
I20260620 23:02:35.596705 18252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:35.596701 18257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:35.599615 18259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8114cfbce722414a981bdaa62faa4b4a. Latest consensus state: current_term: 1 leader_uuid: "8114cfbce722414a981bdaa62faa4b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } } }
I20260620 23:02:35.599614 18258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8114cfbce722414a981bdaa62faa4b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } } }
I20260620 23:02:35.600070 18259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:35.600067 18258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:35.602296 18266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:35.608917 18266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:35.617865 18266 catalog_manager.cc:1347] Generated new cluster ID: 2f5573fbdc974d22ae99d29bd5123ca2
I20260620 23:02:35.618003 18266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:35.626706 18266 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:35.628365 18266 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:35.640151 18266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Generated new TSK 0
I20260620 23:02:35.641151 18266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:35.645632 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:33929
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260620 23:02:35.967584 18276 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260620 23:02:35.967962 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:35.968075 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:35.968205 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:35.975479 18276 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 23:02:35.975580 18276 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 23:02:35.975708 18276 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:35.975811 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:35.975977 18276 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260620 23:02:35.976063 18276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:35.985078 18276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:33929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:35.986486 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:35.988202 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:36.001035 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:36.000972 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:36.000933 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:36.001904 18276 server_base.cc:1034] running on GCE node
I20260620 23:02:36.002540 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:36.003417 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:36.004606 18276 hybrid_clock.cc:648] HybridClock initialized: now 1781996556004550 us; error 46 us; skew 500 ppm
I20260620 23:02:36.004935 18276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:36.007083 18276 webserver.cc:460] Webserver started at http://127.10.134.129:33721/ using document root <none> and password file <none>
I20260620 23:02:36.007870 18276 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:36.007982 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:36.008296 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:36.010883 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4490343f594f412289f880c7de2beec3"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.011629 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4490343f594f412289f880c7de2beec3"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.016309 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 23:02:36.019414 18290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.020617 18276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:02:36.020792 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4490343f594f412289f880c7de2beec3"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.020994 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:36.035871 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:36.036614 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:36.036891 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:36.037766 18276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:36.039139 18276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:36.039290 18276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.039417 18276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:36.039502 18276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.070420 18276 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:38171
I20260620 23:02:36.070469 18402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:38171 every 8 connection(s)
I20260620 23:02:36.071775 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:36.077152 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18276
I20260620 23:02:36.077344 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:36.080992 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:33929
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260620 23:02:36.086781 18403 heartbeater.cc:347] Connected to a master server at 127.10.134.190:33929
I20260620 23:02:36.087091 18403 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:36.087886 18403 heartbeater.cc:511] Master 127.10.134.190:33929 requested a full tablet report, sending...
I20260620 23:02:36.089768 18217 ts_manager.cc:194] Registered new tserver with Master: 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:36.091311 18217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:41879
W20260620 23:02:36.397383 18406 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260620 23:02:36.397727 18406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:36.397825 18406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:36.397969 18406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:36.405143 18406 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 23:02:36.405225 18406 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 23:02:36.405325 18406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:36.405426 18406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:36.405607 18406 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260620 23:02:36.405686 18406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:36.414892 18406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:33929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:36.416163 18406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:36.417871 18406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:36.431337 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:36.431408 18412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:36.431427 18411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:36.435442 18406 server_base.cc:1034] running on GCE node
I20260620 23:02:36.436019 18406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:36.436920 18406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:36.438144 18406 hybrid_clock.cc:648] HybridClock initialized: now 1781996556438102 us; error 58 us; skew 500 ppm
I20260620 23:02:36.438457 18406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:36.440653 18406 webserver.cc:460] Webserver started at http://127.10.134.130:34625/ using document root <none> and password file <none>
I20260620 23:02:36.441373 18406 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:36.441504 18406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:36.441828 18406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:36.444491 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "09ead724cd23457cbe47304c565c1155"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.445242 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "09ead724cd23457cbe47304c565c1155"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.451249 18406 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 23:02:36.454663 18420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.456019 18406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:36.456233 18406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "09ead724cd23457cbe47304c565c1155"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.456483 18406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:36.482192 18406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:36.483049 18406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:36.483346 18406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:36.484274 18406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:36.485684 18406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:36.485810 18406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.485960 18406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:36.486052 18406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 23:02:36.518503 18406 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:42807
I20260620 23:02:36.518592 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:42807 every 8 connection(s)
I20260620 23:02:36.519783 18406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:36.525229 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18406
I20260620 23:02:36.525413 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:36.528695 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:33929
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260620 23:02:36.534955 18533 heartbeater.cc:347] Connected to a master server at 127.10.134.190:33929
I20260620 23:02:36.535265 18533 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:36.535961 18533 heartbeater.cc:511] Master 127.10.134.190:33929 requested a full tablet report, sending...
I20260620 23:02:36.537271 18217 ts_manager.cc:194] Registered new tserver with Master: 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807)
I20260620 23:02:36.538532 18217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:37125
W20260620 23:02:36.845858 18536 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260620 23:02:36.846240 18536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:36.846356 18536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:36.846499 18536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:36.853474 18536 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 23:02:36.853550 18536 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 23:02:36.853655 18536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:36.853740 18536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:36.853916 18536 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260620 23:02:36.853984 18536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:36.863076 18536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:33929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:36.864284 18536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:36.865926 18536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:36.878540 18541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:36.878634 18542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:36.878659 18544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:36.879662 18536 server_base.cc:1034] running on GCE node
I20260620 23:02:36.880239 18536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:36.881096 18536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:36.882349 18536 hybrid_clock.cc:648] HybridClock initialized: now 1781996556882299 us; error 58 us; skew 500 ppm
I20260620 23:02:36.882705 18536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:36.884778 18536 webserver.cc:460] Webserver started at http://127.10.134.131:42443/ using document root <none> and password file <none>
I20260620 23:02:36.885454 18536 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:36.885579 18536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:36.885880 18536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:36.888303 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3c46a2f7e820489aa59bf4b6e65b361c"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.889058 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3c46a2f7e820489aa59bf4b6e65b361c"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.894383 18536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:02:36.897642 18550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.898931 18536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:36.899107 18536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3c46a2f7e820489aa59bf4b6e65b361c"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:36.899312 18536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:36.917099 18536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:36.917906 18536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:36.918272 18536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:36.919166 18536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:36.920584 18536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:36.920706 18536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.920843 18536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:36.920936 18536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:36.953094 18536 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:42843
I20260620 23:02:36.953143 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:42843 every 8 connection(s)
I20260620 23:02:36.954602 18536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:36.964068 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18536
I20260620 23:02:36.964243 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:36.968358 18663 heartbeater.cc:347] Connected to a master server at 127.10.134.190:33929
I20260620 23:02:36.968668 18663 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:36.969311 18663 heartbeater.cc:511] Master 127.10.134.190:33929 requested a full tablet report, sending...
I20260620 23:02:36.970620 18217 ts_manager.cc:194] Registered new tserver with Master: 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843)
I20260620 23:02:36.971382 18217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:49777
I20260620 23:02:36.979621 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:36.994485 18217 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42342:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260620 23:02:37.031342 18338 tablet_service.cc:1467] Processing CreateTablet for tablet db4ad337246546be8e5a83df52a4a3f3 (DEFAULT_TABLE table=TestTable [id=80abc195fdd748109256aac4a30586d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:37.032881 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ad337246546be8e5a83df52a4a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:37.037828 18468 tablet_service.cc:1467] Processing CreateTablet for tablet db4ad337246546be8e5a83df52a4a3f3 (DEFAULT_TABLE table=TestTable [id=80abc195fdd748109256aac4a30586d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:37.039343 18468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ad337246546be8e5a83df52a4a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:37.039886 18598 tablet_service.cc:1467] Processing CreateTablet for tablet db4ad337246546be8e5a83df52a4a3f3 (DEFAULT_TABLE table=TestTable [id=80abc195fdd748109256aac4a30586d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:37.040915 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ad337246546be8e5a83df52a4a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:37.047528 18681 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Bootstrap starting.
I20260620 23:02:37.051102 18681 tablet_bootstrap.cc:654] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:37.051682 18682 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Bootstrap starting.
I20260620 23:02:37.052609 18681 log.cc:824] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:37.055197 18683 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Bootstrap starting.
I20260620 23:02:37.055495 18682 tablet_bootstrap.cc:654] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:37.056870 18682 log.cc:824] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:37.058349 18681 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: No bootstrap required, opened a new log
I20260620 23:02:37.058480 18683 tablet_bootstrap.cc:654] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:37.058624 18681 ts_tablet_manager.cc:1397] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20260620 23:02:37.059376 18683 log.cc:824] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:37.059540 18682 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: No bootstrap required, opened a new log
I20260620 23:02:37.059785 18682 ts_tablet_manager.cc:1397] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 23:02:37.061642 18683 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: No bootstrap required, opened a new log
I20260620 23:02:37.061882 18683 ts_tablet_manager.cc:1397] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 23:02:37.069085 18683 raft_consensus.cc:348] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.069110 18681 raft_consensus.cc:348] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.069437 18683 raft_consensus.cc:374] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:37.069562 18681 raft_consensus.cc:374] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:37.069547 18683 raft_consensus.cc:729] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c46a2f7e820489aa59bf4b6e65b361c, State: Initialized, Role: FOLLOWER
I20260620 23:02:37.069773 18681 raft_consensus.cc:729] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4490343f594f412289f880c7de2beec3, State: Initialized, Role: FOLLOWER
I20260620 23:02:37.070084 18682 raft_consensus.cc:348] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.070348 18683 consensus_queue.cc:260] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.070468 18681 consensus_queue.cc:260] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.070731 18682 raft_consensus.cc:374] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:37.070870 18682 raft_consensus.cc:729] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ead724cd23457cbe47304c565c1155, State: Initialized, Role: FOLLOWER
I20260620 23:02:37.071693 18682 consensus_queue.cc:260] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.076038 18533 heartbeater.cc:503] Master 127.10.134.190:33929 was elected leader, sending a full tablet report...
I20260620 23:02:37.077270 18682 ts_tablet_manager.cc:1428] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Time spent starting tablet: real 0.017s user 0.005s sys 0.010s
I20260620 23:02:37.078148 18663 heartbeater.cc:503] Master 127.10.134.190:33929 was elected leader, sending a full tablet report...
I20260620 23:02:37.079978 18683 ts_tablet_manager.cc:1428] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Time spent starting tablet: real 0.018s user 0.003s sys 0.010s
I20260620 23:02:37.081131 18403 heartbeater.cc:503] Master 127.10.134.190:33929 was elected leader, sending a full tablet report...
I20260620 23:02:37.081845 18681 ts_tablet_manager.cc:1428] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Time spent starting tablet: real 0.023s user 0.016s sys 0.003s
W20260620 23:02:37.122366 18534 tablet.cc:2369] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:02:37.156299 18664 tablet.cc:2369] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:02:37.175357 18404 tablet.cc:2369] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:37.277263 18688 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:37.277616 18688 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.278986 18687 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:37.279408 18688 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843), 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807)
I20260620 23:02:37.279270 18687 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.282377 18687 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:37.288669 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "4490343f594f412289f880c7de2beec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" is_pre_election: true
I20260620 23:02:37.289170 18618 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4490343f594f412289f880c7de2beec3 in term 0.
I20260620 23:02:37.289784 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "4490343f594f412289f880c7de2beec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ead724cd23457cbe47304c565c1155" is_pre_election: true
I20260620 23:02:37.290056 18292 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c46a2f7e820489aa59bf4b6e65b361c, 4490343f594f412289f880c7de2beec3; no voters:
I20260620 23:02:37.290408 18488 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4490343f594f412289f880c7de2beec3 in term 0.
I20260620 23:02:37.290514 18688 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:37.290673 18688 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:37.290769 18688 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:37.291829 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ead724cd23457cbe47304c565c1155" is_pre_election: true
I20260620 23:02:37.292021 18488 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c46a2f7e820489aa59bf4b6e65b361c in term 0.
I20260620 23:02:37.292479 18552 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ead724cd23457cbe47304c565c1155, 3c46a2f7e820489aa59bf4b6e65b361c; no voters:
I20260620 23:02:37.292843 18687 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:37.292981 18687 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:37.293099 18687 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:37.292945 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3" is_pre_election: true
I20260620 23:02:37.293896 18688 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.294409 18358 raft_consensus.cc:2379] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c46a2f7e820489aa59bf4b6e65b361c in current term 1: Already voted for candidate 4490343f594f412289f880c7de2beec3 in this term.
I20260620 23:02:37.294649 18688 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [CANDIDATE]: Term 1 election: Requested vote from peers 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843), 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807)
I20260620 23:02:37.295118 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "4490343f594f412289f880c7de2beec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ead724cd23457cbe47304c565c1155"
I20260620 23:02:37.295161 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "4490343f594f412289f880c7de2beec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c46a2f7e820489aa59bf4b6e65b361c"
I20260620 23:02:37.295306 18488 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:37.295764 18687 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.296227 18618 raft_consensus.cc:2379] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4490343f594f412289f880c7de2beec3 in current term 1: Already voted for candidate 3c46a2f7e820489aa59bf4b6e65b361c in this term.
I20260620 23:02:37.296919 18487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ead724cd23457cbe47304c565c1155"
I20260620 23:02:37.296917 18687 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [CANDIDATE]: Term 1 election: Requested vote from peers 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:37.297632 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3"
I20260620 23:02:37.297855 18358 raft_consensus.cc:2379] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3c46a2f7e820489aa59bf4b6e65b361c in current term 1: Already voted for candidate 4490343f594f412289f880c7de2beec3 in this term.
I20260620 23:02:37.298228 18551 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c46a2f7e820489aa59bf4b6e65b361c; no voters: 09ead724cd23457cbe47304c565c1155, 4490343f594f412289f880c7de2beec3
I20260620 23:02:37.298424 18488 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4490343f594f412289f880c7de2beec3 in term 1.
I20260620 23:02:37.298556 18687 raft_consensus.cc:2726] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 23:02:37.298738 18292 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ead724cd23457cbe47304c565c1155, 4490343f594f412289f880c7de2beec3; no voters: 3c46a2f7e820489aa59bf4b6e65b361c
I20260620 23:02:37.298982 18688 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:37.299386 18688 raft_consensus.cc:686] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 LEADER]: Becoming Leader. State: Replica: 4490343f594f412289f880c7de2beec3, State: Running, Role: LEADER
I20260620 23:02:37.299845 18688 consensus_queue.cc:237] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:37.303732 18217 catalog_manager.cc:5495] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4490343f594f412289f880c7de2beec3 (127.10.134.129). New cstate: current_term: 1 leader_uuid: "4490343f594f412289f880c7de2beec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } } }
I20260620 23:02:37.313226 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:37.315166 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4490343f594f412289f880c7de2beec3 to finish bootstrapping
I20260620 23:02:37.324410 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 09ead724cd23457cbe47304c565c1155 to finish bootstrapping
I20260620 23:02:37.332643 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3c46a2f7e820489aa59bf4b6e65b361c to finish bootstrapping
I20260620 23:02:37.359730 10778 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260620 23:02:37.462894 10778 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260620 23:02:37.666203 10778 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260620 23:02:37.784439 18688 consensus_queue.cc:1035] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:37.791949 18694 consensus_queue.cc:1035] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:38.083426 10778 test_util.cc:268] Using random seed: 1742071077
I20260620 23:02:38.390928 10778 test_util.cc:268] Using random seed: 1742378597
I20260620 23:02:38.797647 10778 test_util.cc:268] Using random seed: 1742785290
I20260620 23:02:39.290867 18704 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4490343f594f412289f880c7de2beec3)
I20260620 23:02:39.291087 18704 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:39.291744 18704 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:39.292369 18488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "09ead724cd23457cbe47304c565c1155" is_pre_election: true
I20260620 23:02:39.294407 18705 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4490343f594f412289f880c7de2beec3)
I20260620 23:02:39.294548 18705 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:39.295867 18705 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:39.302295 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "09ead724cd23457cbe47304c565c1155" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" is_pre_election: true
I20260620 23:02:39.302495 18618 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ead724cd23457cbe47304c565c1155 in term 1.
I20260620 23:02:39.302973 18422 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ead724cd23457cbe47304c565c1155, 3c46a2f7e820489aa59bf4b6e65b361c; no voters:
I20260620 23:02:39.303370 18705 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:02:39.303481 18705 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4490343f594f412289f880c7de2beec3)
I20260620 23:02:39.303618 18705 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:39.305128 10778 test_util.cc:268] Using random seed: 1743292813
I20260620 23:02:39.305918 18705 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:39.306517 18705 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 2 election: Requested vote from peers 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:39.307004 18618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "09ead724cd23457cbe47304c565c1155" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c46a2f7e820489aa59bf4b6e65b361c"
I20260620 23:02:39.307205 18618 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:39.309032 18618 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ead724cd23457cbe47304c565c1155 in term 2.
I20260620 23:02:39.309387 18422 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ead724cd23457cbe47304c565c1155, 3c46a2f7e820489aa59bf4b6e65b361c; no voters:
I20260620 23:02:39.309689 18705 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:02:39.310184 18705 raft_consensus.cc:686] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 LEADER]: Becoming Leader. State: Replica: 09ead724cd23457cbe47304c565c1155, State: Running, Role: LEADER
I20260620 23:02:39.310617 18705 consensus_queue.cc:237] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:39.313583 18217 catalog_manager.cc:5495] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 reported cstate change: term changed from 1 to 2, leader changed from 4490343f594f412289f880c7de2beec3 (127.10.134.129) to 09ead724cd23457cbe47304c565c1155 (127.10.134.130). New cstate: current_term: 2 leader_uuid: "09ead724cd23457cbe47304c565c1155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } } }
I20260620 23:02:39.408951 10778 test_util.cc:268] Using random seed: 1743396607
I20260620 23:02:39.428333 10778 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
I20260620 23:02:39.430073 10778 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20260620 23:02:39.452301 18618 raft_consensus.cc:1261] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Refusing update from remote peer 09ead724cd23457cbe47304c565c1155: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 23:02:39.453198 18709 consensus_queue.cc:1035] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 23:02:39.531363 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3): 1840944 bytes log retention, and flush 2998537 bytes memory
I20260620 23:02:39.565642 18664 maintenance_manager.cc:419] P 3c46a2f7e820489aa59bf4b6e65b361c: Scheduling FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3): 2498485 bytes log retention, and flush 4038921 bytes memory
I20260620 23:02:39.641428 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.109s user 0.052s sys 0.032s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":6275,"dirs.queue_time_us":16918,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":5138,"drs_written":1,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1060,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":14,"thread_start_us":23568,"threads_started":2}
I20260620 23:02:39.643111 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling LogGCOp(db4ad337246546be8e5a83df52a4a3f3): free 1840944 bytes of WAL
I20260620 23:02:39.654237 18425 log_reader.cc:386] T db4ad337246546be8e5a83df52a4a3f3: removed 2 log segments from log reader
I20260620 23:02:39.654459 18425 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000001 (ops 1-9)
I20260620 23:02:39.654639 18425 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000002 (ops 10-16)
I20260620 23:02:39.655570 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: LogGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20260620 23:02:39.656081 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3): 2761368 bytes log retention, and flush 4038921 bytes memory
I20260620 23:02:39.658205 10778 raft_consensus-itest-base.cc:274] Waiting for log GC on 09ead724cd23457cbe47304c565c1155
I20260620 23:02:39.658631 10778 raft_consensus-itest-base.cc:279] Log GC complete on 09ead724cd23457cbe47304c565c1155
I20260620 23:02:39.734396 18555 maintenance_manager.cc:643] P 3c46a2f7e820489aa59bf4b6e65b361c: FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.168s user 0.079s sys 0.025s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":40689,"dirs.queue_time_us":13203,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":783,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1026,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":20,"thread_start_us":42863,"threads_started":2}
I20260620 23:02:39.736295 18664 maintenance_manager.cc:419] P 3c46a2f7e820489aa59bf4b6e65b361c: Scheduling LogGCOp(db4ad337246546be8e5a83df52a4a3f3): free 2498485 bytes of WAL
I20260620 23:02:39.736779 18555 log_reader.cc:386] T db4ad337246546be8e5a83df52a4a3f3: removed 4 log segments from log reader
I20260620 23:02:39.737052 18555 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000001 (ops 1-6)
I20260620 23:02:39.737257 18555 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000002 (ops 7-12)
I20260620 23:02:39.737406 18555 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000003 (ops 13-16)
I20260620 23:02:39.737530 18555 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000004 (ops 17-21)
I20260620 23:02:39.738698 18555 maintenance_manager.cc:643] P 3c46a2f7e820489aa59bf4b6e65b361c: LogGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.002s user 0.000s sys 0.001s Metrics: {}
I20260620 23:02:39.739113 18664 maintenance_manager.cc:419] P 3c46a2f7e820489aa59bf4b6e65b361c: Scheduling FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3): 1577890 bytes log retention, and flush 2998537 bytes memory
I20260620 23:02:39.742370 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.086s user 0.069s sys 0.000s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":955,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":23}
I20260620 23:02:39.743794 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling LogGCOp(db4ad337246546be8e5a83df52a4a3f3): free 920418 bytes of WAL
I20260620 23:02:39.744153 18425 log_reader.cc:386] T db4ad337246546be8e5a83df52a4a3f3: removed 1 log segments from log reader
I20260620 23:02:39.744278 18425 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000003 (ops 17-23)
I20260620 23:02:39.744781 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: LogGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260620 23:02:39.745173 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling CompactRowSetsOp(db4ad337246546be8e5a83df52a4a3f3): perf score=0.981908
I20260620 23:02:39.794658 18555 maintenance_manager.cc:643] P 3c46a2f7e820489aa59bf4b6e65b361c: FlushMRSOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.055s user 0.050s sys 0.005s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1224,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":977,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":17}
I20260620 23:02:39.796221 18664 maintenance_manager.cc:419] P 3c46a2f7e820489aa59bf4b6e65b361c: Scheduling CompactRowSetsOp(db4ad337246546be8e5a83df52a4a3f3): perf score=0.981900
I20260620 23:02:39.881268 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: CompactRowSetsOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.136s user 0.114s sys 0.019s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264314,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":1726,"dirs.run_cpu_time_us":961,"dirs.run_wall_time_us":4698,"drs_written":1,"lbm_read_time_us":1287,"lbm_reads_lt_1ms":54,"lbm_write_time_us":994,"lbm_writes_lt_1ms":23,"mutex_wait_us":175,"num_input_rowsets":2,"peak_mem_usage":186,"rows_written":37,"thread_start_us":19318,"threads_started":2}
I20260620 23:02:39.882773 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3): 549 bytes on disk
I20260620 23:02:39.883750 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20260620 23:02:39.922669 18555 maintenance_manager.cc:643] P 3c46a2f7e820489aa59bf4b6e65b361c: CompactRowSetsOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.126s user 0.101s sys 0.023s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264438,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":1845,"dirs.run_cpu_time_us":814,"dirs.run_wall_time_us":3905,"drs_written":1,"lbm_read_time_us":1399,"lbm_reads_lt_1ms":54,"lbm_write_time_us":1027,"lbm_writes_lt_1ms":23,"mutex_wait_us":62,"num_input_rowsets":2,"peak_mem_usage":257,"rows_written":37,"thread_start_us":8068,"threads_started":3}
I20260620 23:02:39.923847 18664 maintenance_manager.cc:419] P 3c46a2f7e820489aa59bf4b6e65b361c: Scheduling UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3): 549 bytes on disk
I20260620 23:02:39.924742 18555 maintenance_manager.cc:643] P 3c46a2f7e820489aa59bf4b6e65b361c: UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20260620 23:02:41.277422 18421 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.10.134.129:38171 for RPC RequestConsensusVote timed out after 1.980s (ON_OUTBOUND_QUEUE)
W20260620 23:02:41.503960 18551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.10.134.129:38171 timed out after 2.210s (SENT)
I20260620 23:02:41.660214 10778 raft_consensus-itest-base.cc:295] Servers converged with original term 2
I20260620 23:02:41.660351 10778 raft_consensus-itest-base.cc:300] Resuming 4490343f594f412289f880c7de2beec3
I20260620 23:02:41.663622 18357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3" is_pre_election: true
I20260620 23:02:41.664249 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3" is_pre_election: true
I20260620 23:02:41.667362 18488 raft_consensus.cc:1226] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 LEADER]: Rejecting Update request from peer 4490343f594f412289f880c7de2beec3 for earlier term 1. Current term is 2. Ops: []
I20260620 23:02:41.668519 18616 raft_consensus.cc:1226] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Rejecting Update request from peer 4490343f594f412289f880c7de2beec3 for earlier term 1. Current term is 2. Ops: []
I20260620 23:02:41.668609 18694 consensus_queue.cc:1046] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 39, Time since last communication: 0.000s
I20260620 23:02:41.668936 18694 consensus_queue.cc:1225] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [LEADER]: Peer 09ead724cd23457cbe47304c565c1155 log is divergent from this leader: its last log entry 2.39 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 39
I20260620 23:02:41.669077 18694 raft_consensus.cc:3032] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:41.669188 18694 raft_consensus.cc:729] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4490343f594f412289f880c7de2beec3, State: Running, Role: LEADER
I20260620 23:02:41.669456 18694 consensus_queue.cc:260] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } }
I20260620 23:02:41.669999 18694 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:41.670153 18551 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c46a2f7e820489aa59bf4b6e65b361c; no voters: 09ead724cd23457cbe47304c565c1155, 4490343f594f412289f880c7de2beec3
I20260620 23:02:41.675199 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "09ead724cd23457cbe47304c565c1155" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3"
I20260620 23:02:41.675258 18355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "09ead724cd23457cbe47304c565c1155" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3" is_pre_election: true
I20260620 23:02:41.677763 18753 raft_consensus.cc:2726] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:02:41.678462 18351 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ead724cd23457cbe47304c565c1155 in term 2.
I20260620 23:02:41.679021 18354 raft_consensus.cc:1261] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: Refusing update from remote peer 09ead724cd23457cbe47304c565c1155: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 23:02:41.680069 18751 consensus_queue.cc:1035] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260620 23:02:41.680590 18751 consensus_queue.cc:740] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: The logs necessary to catch up peer 4490343f594f412289f880c7de2beec3 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..32: Segment 1 which contained index 2 has been GCed)
I20260620 23:02:41.682201 18751 raft_consensus.cc:998] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Attempting to remove follower 4490343f594f412289f880c7de2beec3 from the Raft config. Reason: The logs necessary to catch up peer 4490343f594f412289f880c7de2beec3 have been garbage collected. The replica will never be able to catch up
I20260620 23:02:41.683243 18751 consensus_queue.cc:237] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } }
I20260620 23:02:41.685097 18616 raft_consensus.cc:1261] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Refusing update from remote peer 09ead724cd23457cbe47304c565c1155: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20260620 23:02:41.685609 18754 consensus_queue.cc:1035] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20260620 23:02:41.688115 18751 raft_consensus.cc:2932] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index -1 to 40, VOTER 4490343f594f412289f880c7de2beec3 (127.10.134.129) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } }
I20260620 23:02:41.688608 18616 raft_consensus.cc:2932] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index -1 to 40, VOTER 4490343f594f412289f880c7de2beec3 (127.10.134.129) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } }
I20260620 23:02:41.690990 18534 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling LogGCOp(db4ad337246546be8e5a83df52a4a3f3): free 1840950 bytes of WAL
I20260620 23:02:41.691293 18425 log_reader.cc:386] T db4ad337246546be8e5a83df52a4a3f3: removed 2 log segments from log reader
I20260620 23:02:41.691393 18425 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000004 (ops 24-30)
I20260620 23:02:41.691474 18425 log.cc:1076] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Deleting log segment in path: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/db4ad337246546be8e5a83df52a4a3f3/wal-000000005 (ops 31-37)
I20260620 23:02:41.692199 18425 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: LogGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20260620 23:02:41.692781 18217 catalog_manager.cc:5495] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c reported cstate change: config changed from index -1 to 40, VOTER 4490343f594f412289f880c7de2beec3 (127.10.134.129) evicted. New cstate: current_term: 2 leader_uuid: "09ead724cd23457cbe47304c565c1155" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } }
I20260620 23:02:41.697921 18338 tablet_service.cc:1514] Processing DeleteTablet for tablet db4ad337246546be8e5a83df52a4a3f3 with delete_type TABLET_DATA_TOMBSTONED (TS 4490343f594f412289f880c7de2beec3 not found in new config with opid_index 40) from {username='slave'} at 127.0.0.1:45344
I20260620 23:02:41.699558 18761 tablet_replica.cc:330] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: stopping tablet replica
I20260620 23:02:41.699929 18761 raft_consensus.cc:2229] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:41.700245 18761 raft_consensus.cc:2258] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:41.701436 18761 ts_tablet_manager.cc:1905] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:41.706164 18761 ts_tablet_manager.cc:1918] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:41.706336 18761 log.cc:1196] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/db4ad337246546be8e5a83df52a4a3f3
I20260620 23:02:41.707173 18201 catalog_manager.cc:4841] TS 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171): tablet db4ad337246546be8e5a83df52a4a3f3 (table TestTable [id=80abc195fdd748109256aac4a30586d9]) successfully deleted
I20260620 23:02:41.714558 18398 generic_service.cc:175] {username='slave'} at 127.0.0.1:45348 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20260620 23:02:41.716321 18488 consensus_queue.cc:237] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:41.718261 18616 raft_consensus.cc:1261] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Refusing update from remote peer 09ead724cd23457cbe47304c565c1155: Log matching property violated. Preceding OpId in replica: term: 2 index: 40. Preceding OpId from leader: term: 2 index: 41. (index mismatch)
W20260620 23:02:41.718613 18421 consensus_peers.cc:487] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 -> Peer 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171): Couldn't send request to peer 4490343f594f412289f880c7de2beec3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:41.718807 18751 consensus_queue.cc:1035] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20260620 23:02:41.720729 18754 raft_consensus.cc:2932] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 LEADER]: Committing config change with OpId 2.41: config changed from index 40 to 41, VOTER 4490343f594f412289f880c7de2beec3 (127.10.134.129) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } } }
I20260620 23:02:41.721285 18616 raft_consensus.cc:2932] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c [term 2 FOLLOWER]: Committing config change with OpId 2.41: config changed from index 40 to 41, VOTER 4490343f594f412289f880c7de2beec3 (127.10.134.129) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } } }
I20260620 23:02:41.725669 18217 catalog_manager.cc:5495] T db4ad337246546be8e5a83df52a4a3f3 P 3c46a2f7e820489aa59bf4b6e65b361c reported cstate change: config changed from index 40 to 41, VOTER 4490343f594f412289f880c7de2beec3 (127.10.134.129) added. New cstate: current_term: 2 leader_uuid: "09ead724cd23457cbe47304c565c1155" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } } }
I20260620 23:02:42.246477 18763 ts_tablet_manager.cc:927] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Initiating tablet copy from peer 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807)
I20260620 23:02:42.247478 18763 tablet_copy_client.cc:316] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet copy: Beginning tablet copy session from remote peer at address 127.10.134.130:42807
I20260620 23:02:42.248955 18508 tablet_copy_service.cc:140] P 09ead724cd23457cbe47304c565c1155: Received BeginTabletCopySession request for tablet db4ad337246546be8e5a83df52a4a3f3 from peer 4490343f594f412289f880c7de2beec3 ({username='slave'} at 127.10.134.129:45027)
I20260620 23:02:42.249195 18508 tablet_copy_service.cc:161] P 09ead724cd23457cbe47304c565c1155: Beginning new tablet copy session on tablet db4ad337246546be8e5a83df52a4a3f3 from peer 4490343f594f412289f880c7de2beec3 at {username='slave'} at 127.10.134.129:45027: session id = 4490343f594f412289f880c7de2beec3-db4ad337246546be8e5a83df52a4a3f3
I20260620 23:02:42.252336 18508 tablet_copy_source_session.cc:205] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Tablet Copy: opened 5 blocks and 1 log segments
I20260620 23:02:42.254536 18763 ts_tablet_manager.cc:1905] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260620 23:02:42.259199 18763 ts_tablet_manager.cc:1918] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260620 23:02:42.259591 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ad337246546be8e5a83df52a4a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 23:02:42.262115 18763 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260620 23:02:42.262215 18763 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260620 23:02:42.262279 18763 fault_injection.cc:59] injecting fault for kudu (pid 18276): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20260620 23:02:42.386152 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 18406
I20260620 23:02:42.387441 18528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:42.506940 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 18406
I20260620 23:02:42.530632 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 18536
I20260620 23:02:42.531729 18658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:42.661597 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 18536
I20260620 23:02:42.683840 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 18186
I20260620 23:02:42.685006 18247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:42.789731 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 18186
I20260620 23:02:42.813403 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:33929
--webserver_interface=127.10.134.190
--webserver_port=46603
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:33929
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260620 23:02:43.130661 18768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:43.130934 18768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:43.131090 18768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:43.138029 18768 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:43.138163 18768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:43.138233 18768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:43.138350 18768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:43.138401 18768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:43.147059 18768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:33929
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:33929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=46603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:43.148260 18768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:43.150009 18768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:43.161314 18773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:43.161351 18776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:43.161248 18774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:43.162228 18768 server_base.cc:1034] running on GCE node
I20260620 23:02:43.163014 18768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:43.164354 18768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:43.165581 18768 hybrid_clock.cc:648] HybridClock initialized: now 1781996563165546 us; error 60 us; skew 500 ppm
I20260620 23:02:43.165923 18768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:43.168335 18768 webserver.cc:460] Webserver started at http://127.10.134.190:46603/ using document root <none> and password file <none>
I20260620 23:02:43.169009 18768 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:43.169155 18768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:43.174121 18768 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:02:43.176918 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:43.178208 18768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:43.178387 18768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8114cfbce722414a981bdaa62faa4b4a"
format_stamp: "Formatted at 2026-06-20 23:02:35 on dist-test-slave-8k25"
I20260620 23:02:43.179298 18768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:43.196764 18768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:43.197669 18768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:43.198009 18768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:43.215456 18768 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:33929
I20260620 23:02:43.215494 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:33929 every 8 connection(s)
I20260620 23:02:43.216902 18768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:43.223124 18834 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:02:43.223780 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18768
I20260620 23:02:43.224571 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:42807
--local_ip_for_outbound_sockets=127.10.134.130
--tserver_master_addrs=127.10.134.190:33929
--webserver_port=34625
--webserver_interface=127.10.134.130
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260620 23:02:43.226528 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Bootstrap starting.
I20260620 23:02:43.248059 18834 log.cc:824] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:43.263586 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=5 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:02:43.264252 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Bootstrap complete.
I20260620 23:02:43.276115 18834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:43.276961 18834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8114cfbce722414a981bdaa62faa4b4a, State: Initialized, Role: FOLLOWER
I20260620 23:02:43.277604 18834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:43.277837 18834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:43.277981 18834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:43.278180 18834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:43.281898 18834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:43.282524 18834 leader_election.cc:302] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8114cfbce722414a981bdaa62faa4b4a; no voters:
I20260620 23:02:43.283229 18834 leader_election.cc:288] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:02:43.283422 18838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:02:43.285168 18834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:43.290313 18838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [term 2 LEADER]: Becoming Leader. State: Replica: 8114cfbce722414a981bdaa62faa4b4a, State: Running, Role: LEADER
I20260620 23:02:43.290853 18838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } }
I20260620 23:02:43.293553 18840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8114cfbce722414a981bdaa62faa4b4a. Latest consensus state: current_term: 2 leader_uuid: "8114cfbce722414a981bdaa62faa4b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } } }
I20260620 23:02:43.293547 18839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8114cfbce722414a981bdaa62faa4b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8114cfbce722414a981bdaa62faa4b4a" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 33929 } } }
I20260620 23:02:43.294026 18839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:43.296192 18840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:43.300724 18852 catalog_manager.cc:1259] Loaded cluster ID: 2f5573fbdc974d22ae99d29bd5123ca2
I20260620 23:02:43.300822 18852 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: loading cluster ID for follower catalog manager: success
I20260620 23:02:43.303529 18852 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: acquiring CA information for follower catalog manager: success
I20260620 23:02:43.305208 18852 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:02:43.305380 18853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:43.307454 18853 catalog_manager.cc:669] Loaded metadata for table TestTable [id=80abc195fdd748109256aac4a30586d9]
I20260620 23:02:43.308987 18853 tablet_loader.cc:96] loaded metadata for tablet db4ad337246546be8e5a83df52a4a3f3 (table TestTable [id=80abc195fdd748109256aac4a30586d9])
I20260620 23:02:43.309293 18853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:43.310205 18853 catalog_manager.cc:1259] Loaded cluster ID: 2f5573fbdc974d22ae99d29bd5123ca2
I20260620 23:02:43.310293 18853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:43.311586 18853 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:43.312409 18853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8114cfbce722414a981bdaa62faa4b4a: Loaded TSK: 0
I20260620 23:02:43.313124 18853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:02:43.584897 18836 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260620 23:02:43.585276 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:43.585381 18836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:43.585503 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:43.593029 18836 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 23:02:43.593106 18836 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 23:02:43.593210 18836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:43.593283 18836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:43.593493 18836 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260620 23:02:43.593549 18836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:43.602555 18836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:42807
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=34625
--tserver_master_addrs=127.10.134.190:33929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:43.603888 18836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:43.605713 18836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:43.619061 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:43.619084 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:43.623136 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:43.623492 18836 server_base.cc:1034] running on GCE node
I20260620 23:02:43.624199 18836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:43.625324 18836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:43.626552 18836 hybrid_clock.cc:648] HybridClock initialized: now 1781996563626483 us; error 61 us; skew 500 ppm
I20260620 23:02:43.626915 18836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:43.629526 18836 webserver.cc:460] Webserver started at http://127.10.134.130:34625/ using document root <none> and password file <none>
I20260620 23:02:43.630426 18836 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:43.630558 18836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:43.637295 18836 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:02:43.647125 18872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:43.648559 18836 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260620 23:02:43.648731 18836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "09ead724cd23457cbe47304c565c1155"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:43.650002 18836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 5
Total live bytes: 136550
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:43.667162 18836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:43.668061 18836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:43.668358 18836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:43.669458 18836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:43.671660 18879 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:02:43.677479 18836 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:02:43.677577 18836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 23:02:43.677670 18836 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:02:43.680578 18836 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:02:43.680667 18836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260620 23:02:43.681071 18879 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Bootstrap starting.
I20260620 23:02:43.701116 18879 log.cc:824] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:43.715850 18879 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates
I20260620 23:02:43.716423 18879 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Bootstrap complete.
I20260620 23:02:43.716995 18879 ts_tablet_manager.cc:1397] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.012s
I20260620 23:02:43.726598 18879 raft_consensus.cc:348] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:43.727870 18879 raft_consensus.cc:729] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ead724cd23457cbe47304c565c1155, State: Initialized, Role: FOLLOWER
I20260620 23:02:43.728215 18836 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:42807
I20260620 23:02:43.728286 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:42807 every 8 connection(s)
I20260620 23:02:43.728669 18879 consensus_queue.cc:260] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 2.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:43.729575 18836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:43.730559 18879 ts_tablet_manager.cc:1428] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Time spent starting tablet: real 0.013s user 0.014s sys 0.002s
I20260620 23:02:43.730736 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18836
I20260620 23:02:43.731505 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:38171
--local_ip_for_outbound_sockets=127.10.134.129
--tserver_master_addrs=127.10.134.190:33929
--webserver_port=33721
--webserver_interface=127.10.134.129
--builtin_ntp_servers=127.10.134.148:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260620 23:02:43.737458 18989 maintenance_manager.cc:419] P 09ead724cd23457cbe47304c565c1155: Scheduling UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3): 549 bytes on disk
I20260620 23:02:43.738713 18877 maintenance_manager.cc:643] P 09ead724cd23457cbe47304c565c1155: UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20260620 23:02:43.756953 18988 heartbeater.cc:347] Connected to a master server at 127.10.134.190:33929
I20260620 23:02:43.757323 18988 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:43.758232 18988 heartbeater.cc:511] Master 127.10.134.190:33929 requested a full tablet report, sending...
I20260620 23:02:43.761582 18799 ts_manager.cc:194] Registered new tserver with Master: 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807)
I20260620 23:02:43.764257 18799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:52151
I20260620 23:02:43.768261 18988 heartbeater.cc:503] Master 127.10.134.190:33929 was elected leader, sending a full tablet report...
W20260620 23:02:44.056279 18992 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260620 23:02:44.056582 18992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:44.056645 18992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:44.056720 18992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:44.063651 18992 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260620 23:02:44.063727 18992 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260620 23:02:44.063791 18992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260620 23:02:44.063841 18992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:44.063953 18992 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260620 23:02:44.064002 18992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:44.072597 18992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:38171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=33721
--tserver_master_addrs=127.10.134.190:33929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:44.073702 18992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:44.075357 18992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:44.087795 19000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:44.088039 18997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:44.088092 18998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:44.089068 18992 server_base.cc:1034] running on GCE node
I20260620 23:02:44.089694 18992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:44.090762 18992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:44.091982 18992 hybrid_clock.cc:648] HybridClock initialized: now 1781996564091945 us; error 30 us; skew 500 ppm
I20260620 23:02:44.092320 18992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:44.094592 18992 webserver.cc:460] Webserver started at http://127.10.134.129:33721/ using document root <none> and password file <none>
I20260620 23:02:44.095321 18992 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:44.095471 18992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:44.100489 18992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 23:02:44.103221 19006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:44.104605 18992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:44.104784 18992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4490343f594f412289f880c7de2beec3"
format_stamp: "Formatted at 2026-06-20 23:02:36 on dist-test-slave-8k25"
I20260620 23:02:44.105706 18992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:44.139261 18992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:44.140129 18992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:44.140450 18992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:44.141494 18992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:44.143411 19013 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 23:02:44.148855 19013 ts_tablet_manager.cc:1852] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260620 23:02:44.151927 19013 ts_tablet_manager.cc:1905] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:44.158787 19013 ts_tablet_manager.cc:1918] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:44.159816 18992 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 23:02:44.159930 18992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260620 23:02:44.160108 18992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:44.160182 18992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:44.193265 18992 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:38171
I20260620 23:02:44.193311 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:38171 every 8 connection(s)
I20260620 23:02:44.194602 18992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:44.201463 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 18992
I20260620 23:02:44.201751 10778 test_util.cc:268] Using random seed: 1748189444
I20260620 23:02:44.210210 19120 heartbeater.cc:347] Connected to a master server at 127.10.134.190:33929
I20260620 23:02:44.210578 19120 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:44.211419 19120 heartbeater.cc:511] Master 127.10.134.190:33929 requested a full tablet report, sending...
I20260620 23:02:44.213315 18799 ts_manager.cc:194] Registered new tserver with Master: 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:44.214504 18799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:51101
I20260620 23:02:44.220718 10778 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260620 23:02:44.272941 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.272987 18899 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.272836 18900 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.272897 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.277354 19124 meta_cache.cc:302] tablet db4ad337246546be8e5a83df52a4a3f3: replica 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843) has failed: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42843: connect: Connection refused (error 111)
W20260620 23:02:44.294268 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.295198 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.302546 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.302624 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.303517 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.315610 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.316749 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.325361 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.338385 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.353497 18901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.353677 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.366807 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.370942 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.388947 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.398767 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.413743 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.427791 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.439594 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.462407 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.468708 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.500743 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.501609 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.523784 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.534045 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.552743 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.572278 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.592821 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.619308 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.636569 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.654850 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.681547 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.703279 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.717808 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.745203 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.767876 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.795506 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.814460 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.843956 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.870428 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.897141 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.915059 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.948014 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:44.978758 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.014907 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.029202 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.063838 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.096032 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.137135 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.152362 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.197353 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
I20260620 23:02:45.217440 19120 heartbeater.cc:503] Master 127.10.134.190:33929 was elected leader, sending a full tablet report...
W20260620 23:02:45.231022 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
W20260620 23:02:45.268730 18898 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57396: Illegal state: replica 09ead724cd23457cbe47304c565c1155 is not leader of this config: current role FOLLOWER
I20260620 23:02:45.277572 19136 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:45.277899 19136 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:45.279589 19136 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
W20260620 23:02:45.281306 18874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42843: connect: Connection refused (error 111)
W20260620 23:02:45.283187 18874 leader_election.cc:334] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843): Network error: Client connection negotiation failed: client connection to 127.10.134.131:42843: connect: Connection refused (error 111)
I20260620 23:02:45.287830 19075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "09ead724cd23457cbe47304c565c1155" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3" is_pre_election: true
I20260620 23:02:45.290745 19075 raft_consensus.cc:1789] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260620 23:02:45.291496 19075 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ead724cd23457cbe47304c565c1155 in term 2.
I20260620 23:02:45.292061 18873 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ead724cd23457cbe47304c565c1155, 4490343f594f412289f880c7de2beec3; no voters: 3c46a2f7e820489aa59bf4b6e65b361c
I20260620 23:02:45.292433 19136 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:02:45.292604 19136 raft_consensus.cc:482] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:45.292717 19136 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:02:45.296286 19136 raft_consensus.cc:504] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:45.296895 19136 leader_election.cc:288] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 3 election: Requested vote from peers 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843), 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171)
I20260620 23:02:45.297402 19075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ad337246546be8e5a83df52a4a3f3" candidate_uuid: "09ead724cd23457cbe47304c565c1155" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "4490343f594f412289f880c7de2beec3"
I20260620 23:02:45.297639 19075 raft_consensus.cc:1789] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260620 23:02:45.297762 19075 raft_consensus.cc:3037] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 2 FOLLOWER]: Advancing to term 3
W20260620 23:02:45.298442 18874 leader_election.cc:334] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843): Network error: Client connection negotiation failed: client connection to 127.10.134.131:42843: connect: Connection refused (error 111)
I20260620 23:02:45.300034 19075 raft_consensus.cc:2454] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ead724cd23457cbe47304c565c1155 in term 3.
I20260620 23:02:45.300443 18873 leader_election.cc:302] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ead724cd23457cbe47304c565c1155, 4490343f594f412289f880c7de2beec3; no voters: 3c46a2f7e820489aa59bf4b6e65b361c
I20260620 23:02:45.300729 19136 raft_consensus.cc:2781] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:02:45.301134 19136 raft_consensus.cc:686] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [term 3 LEADER]: Becoming Leader. State: Replica: 09ead724cd23457cbe47304c565c1155, State: Running, Role: LEADER
I20260620 23:02:45.301616 19136 consensus_queue.cc:237] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 41, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:45.305721 18790 catalog_manager.cc:5495] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "09ead724cd23457cbe47304c565c1155" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } } }
W20260620 23:02:45.315659 19124 meta_cache.cc:302] tablet db4ad337246546be8e5a83df52a4a3f3: replica 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843) has failed: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42843: connect: Connection refused (error 111)
W20260620 23:02:45.315992 18873 consensus_peers.cc:487] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 -> Peer 4490343f594f412289f880c7de2beec3 (127.10.134.129:38171): Couldn't send request to peer 4490343f594f412289f880c7de2beec3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260620 23:02:45.316696 18874 consensus_peers.cc:487] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155 -> Peer 3c46a2f7e820489aa59bf4b6e65b361c (127.10.134.131:42843): Couldn't send request to peer 3c46a2f7e820489aa59bf4b6e65b361c. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:45.756426 19148 tablet_replica.cc:330] stopping tablet replica
I20260620 23:02:45.756712 19148 raft_consensus.cc:2229] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:45.756912 19148 raft_consensus.cc:2258] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:45.757277 19148 ts_tablet_manager.cc:927] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Initiating tablet copy from peer 09ead724cd23457cbe47304c565c1155 (127.10.134.130:42807)
I20260620 23:02:45.758344 19148 tablet_copy_client.cc:316] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet copy: Beginning tablet copy session from remote peer at address 127.10.134.130:42807
I20260620 23:02:45.766513 18962 tablet_copy_service.cc:140] P 09ead724cd23457cbe47304c565c1155: Received BeginTabletCopySession request for tablet db4ad337246546be8e5a83df52a4a3f3 from peer 4490343f594f412289f880c7de2beec3 ({username='slave'} at 127.10.134.129:37237)
I20260620 23:02:45.766732 18962 tablet_copy_service.cc:161] P 09ead724cd23457cbe47304c565c1155: Beginning new tablet copy session on tablet db4ad337246546be8e5a83df52a4a3f3 from peer 4490343f594f412289f880c7de2beec3 at {username='slave'} at 127.10.134.129:37237: session id = 4490343f594f412289f880c7de2beec3-db4ad337246546be8e5a83df52a4a3f3
I20260620 23:02:45.769800 18962 tablet_copy_source_session.cc:205] T db4ad337246546be8e5a83df52a4a3f3 P 09ead724cd23457cbe47304c565c1155: Tablet Copy: opened 5 blocks and 1 log segments
I20260620 23:02:45.772305 19148 ts_tablet_manager.cc:1905] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260620 23:02:45.776829 19148 ts_tablet_manager.cc:1918] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260620 23:02:45.777323 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ad337246546be8e5a83df52a4a3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:45.779803 19148 tablet_copy_client.cc:791] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet copy: Starting download of 5 data blocks...
I20260620 23:02:45.790112 19148 tablet_copy_client.cc:655] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet copy: Starting download of 1 WAL segments...
I20260620 23:02:45.793802 19148 tablet_copy_client.cc:531] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:02:45.797068 19148 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Bootstrap starting.
I20260620 23:02:45.807663 19148 log.cc:824] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:45.816464 19148 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260620 23:02:45.817081 19148 tablet_bootstrap.cc:492] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Bootstrap complete.
I20260620 23:02:45.817489 19148 ts_tablet_manager.cc:1397] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.001s
I20260620 23:02:45.820370 19148 raft_consensus.cc:348] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:45.822956 19148 raft_consensus.cc:729] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4490343f594f412289f880c7de2beec3, State: Initialized, Role: FOLLOWER
I20260620 23:02:45.823596 19148 consensus_queue.cc:260] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 3.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "3c46a2f7e820489aa59bf4b6e65b361c" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42843 } } peers { permanent_uuid: "09ead724cd23457cbe47304c565c1155" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 42807 } } peers { permanent_uuid: "4490343f594f412289f880c7de2beec3" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38171 } attrs { } }
I20260620 23:02:45.825186 19148 ts_tablet_manager.cc:1428] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260620 23:02:45.826242 18962 tablet_copy_service.cc:342] P 09ead724cd23457cbe47304c565c1155: Request end of tablet copy session 4490343f594f412289f880c7de2beec3-db4ad337246546be8e5a83df52a4a3f3 received from {username='slave'} at 127.10.134.129:37237
I20260620 23:02:45.826422 18962 tablet_copy_service.cc:433] P 09ead724cd23457cbe47304c565c1155: ending tablet copy session 4490343f594f412289f880c7de2beec3-db4ad337246546be8e5a83df52a4a3f3 on tablet db4ad337246546be8e5a83df52a4a3f3 with peer 4490343f594f412289f880c7de2beec3
I20260620 23:02:45.900045 19121 maintenance_manager.cc:419] P 4490343f594f412289f880c7de2beec3: Scheduling UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3): 549 bytes on disk
I20260620 23:02:45.901425 19011 maintenance_manager.cc:643] P 4490343f594f412289f880c7de2beec3: UndoDeltaBlockGCOp(db4ad337246546be8e5a83df52a4a3f3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20260620 23:02:46.254761 19075 raft_consensus.cc:1203] T db4ad337246546be8e5a83df52a4a3f3 P 4490343f594f412289f880c7de2beec3 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 2.41->[3.42-3.46] Dedup: 3.46->[]
I20260620 23:02:47.264377 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 18992
I20260620 23:02:47.272135 19115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:47.376171 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 18992
I20260620 23:02:47.395164 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 18836
I20260620 23:02:47.402206 18982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:47.508741 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 18836
I20260620 23:02:47.528540 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 18768
I20260620 23:02:47.535207 18829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:47.638916 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 18768
2026-06-20T23:02:47Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (12571 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20260620 23:02:47.681438 10778 test_util.cc:268] Using random seed: 1751669128
I20260620 23:02:47.683344 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:47.683434 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:47.683508 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:02:47.683571 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:02:47.683638 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:47Z Disabled control of system clock
I20260620 23:02:47.717749 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:38653
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:38653 with env {}
W20260620 23:02:48.030205 19185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:48.030520 19185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:48.030670 19185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:48.037756 19185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:48.037894 19185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:48.038009 19185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:48.038105 19185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:48.046994 19185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:38653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:48.048273 19185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:48.050081 19185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:48.061414 19190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:48.061472 19191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:48.061451 19193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:48.062620 19185 server_base.cc:1034] running on GCE node
I20260620 23:02:48.063359 19185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:48.064687 19185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:48.065917 19185 hybrid_clock.cc:648] HybridClock initialized: now 1781996568065848 us; error 53 us; skew 500 ppm
I20260620 23:02:48.066286 19185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:48.068502 19185 webserver.cc:460] Webserver started at http://127.10.134.190:39975/ using document root <none> and password file <none>
I20260620 23:02:48.069166 19185 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:48.069314 19185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:48.069603 19185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:48.072160 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "63426e3e89254b15823204c4411cf621"
format_stamp: "Formatted at 2026-06-20 23:02:48 on dist-test-slave-8k25"
I20260620 23:02:48.072913 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "63426e3e89254b15823204c4411cf621"
format_stamp: "Formatted at 2026-06-20 23:02:48 on dist-test-slave-8k25"
I20260620 23:02:48.077800 19185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 23:02:48.081063 19199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:48.082597 19185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:02:48.082787 19185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "63426e3e89254b15823204c4411cf621"
format_stamp: "Formatted at 2026-06-20 23:02:48 on dist-test-slave-8k25"
I20260620 23:02:48.083025 19185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:48.099812 19185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:48.100639 19185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:48.100937 19185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:48.118896 19185 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:38653
I20260620 23:02:48.118939 19250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:38653 every 8 connection(s)
I20260620 23:02:48.120213 19185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:48.123893 19251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:48.127763 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 19185
I20260620 23:02:48.127943 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:48.134891 19251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621: Bootstrap starting.
I20260620 23:02:48.138545 19251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:48.139830 19251 log.cc:824] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:48.142709 19251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621: No bootstrap required, opened a new log
I20260620 23:02:48.149629 19251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63426e3e89254b15823204c4411cf621" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38653 } }
I20260620 23:02:48.150020 19251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:48.150138 19251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63426e3e89254b15823204c4411cf621, State: Initialized, Role: FOLLOWER
I20260620 23:02:48.150745 19251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63426e3e89254b15823204c4411cf621" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38653 } }
I20260620 23:02:48.150971 19251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:48.151136 19251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:48.151311 19251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:48.153252 19251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63426e3e89254b15823204c4411cf621" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38653 } }
I20260620 23:02:48.153729 19251 leader_election.cc:302] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63426e3e89254b15823204c4411cf621; no voters:
I20260620 23:02:48.154328 19251 leader_election.cc:288] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:48.154464 19256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:48.155949 19256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [term 1 LEADER]: Becoming Leader. State: Replica: 63426e3e89254b15823204c4411cf621, State: Running, Role: LEADER
I20260620 23:02:48.156419 19256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63426e3e89254b15823204c4411cf621" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38653 } }
I20260620 23:02:48.156607 19251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:48.158982 19258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63426e3e89254b15823204c4411cf621. Latest consensus state: current_term: 1 leader_uuid: "63426e3e89254b15823204c4411cf621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63426e3e89254b15823204c4411cf621" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38653 } } }
I20260620 23:02:48.159339 19258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:48.159942 19257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63426e3e89254b15823204c4411cf621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63426e3e89254b15823204c4411cf621" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 38653 } } }
I20260620 23:02:48.160336 19257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:48.163334 19264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:48.168326 19264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:48.176877 19264 catalog_manager.cc:1347] Generated new cluster ID: b8d17a19d4fa4ed9a2f0a9b9d6d14d53
I20260620 23:02:48.177006 19264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:48.210520 19264 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:48.212066 19264 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:48.219797 19264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63426e3e89254b15823204c4411cf621: Generated new TSK 0
I20260620 23:02:48.220563 19264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:48.226366 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38653
--builtin_ntp_servers=127.10.134.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20260620 23:02:48.554826 19276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:48.555102 19276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:48.555231 19276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:48.562703 19276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:48.562918 19276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:48.571985 19276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:48.573271 19276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:48.575054 19276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:48.587826 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:48.587882 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:48.588054 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:48.588495 19276 server_base.cc:1034] running on GCE node
I20260620 23:02:48.589056 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:48.589836 19276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:48.591060 19276 hybrid_clock.cc:648] HybridClock initialized: now 1781996568590996 us; error 42 us; skew 500 ppm
I20260620 23:02:48.591341 19276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:48.593539 19276 webserver.cc:460] Webserver started at http://127.10.134.129:37387/ using document root <none> and password file <none>
I20260620 23:02:48.594244 19276 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:48.594367 19276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:48.594666 19276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:48.597090 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e795fe13bb2640f28ba823381e23d206"
format_stamp: "Formatted at 2026-06-20 23:02:48 on dist-test-slave-8k25"
I20260620 23:02:48.597893 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e795fe13bb2640f28ba823381e23d206"
format_stamp: "Formatted at 2026-06-20 23:02:48 on dist-test-slave-8k25"
I20260620 23:02:48.602988 19276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:02:48.606474 19290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:48.607894 19276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 23:02:48.608064 19276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e795fe13bb2640f28ba823381e23d206"
format_stamp: "Formatted at 2026-06-20 23:02:48 on dist-test-slave-8k25"
I20260620 23:02:48.608263 19276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:48.623202 19276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:48.624018 19276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:48.624302 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:48.625259 19276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:48.626796 19276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:48.626904 19276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:48.627053 19276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:48.627123 19276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:48.661518 19276 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:42209
I20260620 23:02:48.661556 19402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:42209 every 8 connection(s)
I20260620 23:02:48.662986 19276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:48.668619 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 19276
I20260620 23:02:48.668838 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:48.672058 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38653
--builtin_ntp_servers=127.10.134.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260620 23:02:48.676676 19403 heartbeater.cc:347] Connected to a master server at 127.10.134.190:38653
I20260620 23:02:48.676985 19403 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:48.677675 19403 heartbeater.cc:511] Master 127.10.134.190:38653 requested a full tablet report, sending...
I20260620 23:02:48.679723 19216 ts_manager.cc:194] Registered new tserver with Master: e795fe13bb2640f28ba823381e23d206 (127.10.134.129:42209)
I20260620 23:02:48.681317 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:35775
W20260620 23:02:48.991272 19407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:48.991539 19407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:48.991698 19407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:48.998659 19407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:48.998876 19407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:49.007718 19407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:49.008966 19407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:49.010680 19407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:49.023373 19412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:49.023398 19413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:49.023367 19415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:49.024268 19407 server_base.cc:1034] running on GCE node
I20260620 23:02:49.024781 19407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:49.025575 19407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:49.026782 19407 hybrid_clock.cc:648] HybridClock initialized: now 1781996569026711 us; error 49 us; skew 500 ppm
I20260620 23:02:49.027031 19407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:49.029080 19407 webserver.cc:460] Webserver started at http://127.10.134.130:38271/ using document root <none> and password file <none>
I20260620 23:02:49.029690 19407 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:49.029841 19407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:49.030215 19407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:49.032635 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ce75845903d6431eb2175c5ec8c70b74"
format_stamp: "Formatted at 2026-06-20 23:02:49 on dist-test-slave-8k25"
I20260620 23:02:49.033306 19407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ce75845903d6431eb2175c5ec8c70b74"
format_stamp: "Formatted at 2026-06-20 23:02:49 on dist-test-slave-8k25"
I20260620 23:02:49.038197 19407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 23:02:49.041327 19421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:49.042703 19407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:49.042834 19407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce75845903d6431eb2175c5ec8c70b74"
format_stamp: "Formatted at 2026-06-20 23:02:49 on dist-test-slave-8k25"
I20260620 23:02:49.042997 19407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:49.056520 19407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:49.057225 19407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:49.057494 19407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:49.058434 19407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:49.059751 19407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:49.059823 19407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:49.059950 19407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:49.060000 19407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:49.092991 19407 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:45365
I20260620 23:02:49.092991 19533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:45365 every 8 connection(s)
I20260620 23:02:49.094340 19407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:49.094748 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 19407
I20260620 23:02:49.094909 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:49.098649 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38653
--builtin_ntp_servers=127.10.134.148:39043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260620 23:02:49.108068 19534 heartbeater.cc:347] Connected to a master server at 127.10.134.190:38653
I20260620 23:02:49.108355 19534 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:49.109064 19534 heartbeater.cc:511] Master 127.10.134.190:38653 requested a full tablet report, sending...
I20260620 23:02:49.110461 19216 ts_manager.cc:194] Registered new tserver with Master: ce75845903d6431eb2175c5ec8c70b74 (127.10.134.130:45365)
I20260620 23:02:49.111672 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:60257
W20260620 23:02:49.426507 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:49.426785 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:49.426867 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:49.433950 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:49.434106 19537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:49.443672 19537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:39043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:38653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:49.444854 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:49.446648 19537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:49.459643 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:49.459661 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:49.459667 19542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:49.460404 19537 server_base.cc:1034] running on GCE node
I20260620 23:02:49.461012 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:49.461881 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:49.463188 19537 hybrid_clock.cc:648] HybridClock initialized: now 1781996569463105 us; error 35 us; skew 500 ppm
I20260620 23:02:49.463603 19537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:49.465910 19537 webserver.cc:460] Webserver started at http://127.10.134.131:34795/ using document root <none> and password file <none>
I20260620 23:02:49.466670 19537 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:49.466795 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:49.467120 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:49.469693 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1271d50966c24c0eb25f634e06944bb7"
format_stamp: "Formatted at 2026-06-20 23:02:49 on dist-test-slave-8k25"
I20260620 23:02:49.470518 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1271d50966c24c0eb25f634e06944bb7"
format_stamp: "Formatted at 2026-06-20 23:02:49 on dist-test-slave-8k25"
I20260620 23:02:49.475508 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:02:49.478806 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:49.480094 19537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:49.480252 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1271d50966c24c0eb25f634e06944bb7"
format_stamp: "Formatted at 2026-06-20 23:02:49 on dist-test-slave-8k25"
I20260620 23:02:49.480434 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:49.498468 19537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:49.499296 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:49.499581 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:49.500519 19537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:49.501909 19537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:49.502035 19537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:49.502200 19537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:49.502250 19537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:49.535254 19537 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:42987
I20260620 23:02:49.535298 19663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:42987 every 8 connection(s)
I20260620 23:02:49.536545 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:49.542239 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 19537
I20260620 23:02:49.542441 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:49.550633 19664 heartbeater.cc:347] Connected to a master server at 127.10.134.190:38653
I20260620 23:02:49.550948 19664 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:49.551641 19664 heartbeater.cc:511] Master 127.10.134.190:38653 requested a full tablet report, sending...
I20260620 23:02:49.552986 19216 ts_manager.cc:194] Registered new tserver with Master: 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987)
I20260620 23:02:49.553761 19216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:34385
I20260620 23:02:49.558161 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:49.573527 19216 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58384:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260620 23:02:49.580233 19216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:49.624625 19599 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b1b9653ca4ed0982c13797f601c60 (DEFAULT_TABLE table=TestTable [id=7f2880cc67a94b8f8baa4a09840065e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:49.626127 19599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b1b9653ca4ed0982c13797f601c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:49.630107 19338 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b1b9653ca4ed0982c13797f601c60 (DEFAULT_TABLE table=TestTable [id=7f2880cc67a94b8f8baa4a09840065e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:49.631487 19338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b1b9653ca4ed0982c13797f601c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:49.636783 19469 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b1b9653ca4ed0982c13797f601c60 (DEFAULT_TABLE table=TestTable [id=7f2880cc67a94b8f8baa4a09840065e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:49.637876 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b1b9653ca4ed0982c13797f601c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:49.641331 19682 tablet_bootstrap.cc:492] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: Bootstrap starting.
I20260620 23:02:49.645279 19682 tablet_bootstrap.cc:654] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:49.645789 19683 tablet_bootstrap.cc:492] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Bootstrap starting.
I20260620 23:02:49.646884 19682 log.cc:824] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:49.649014 19683 tablet_bootstrap.cc:654] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:49.649910 19683 log.cc:824] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:49.652012 19686 tablet_bootstrap.cc:492] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Bootstrap starting.
I20260620 23:02:49.652480 19682 tablet_bootstrap.cc:492] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: No bootstrap required, opened a new log
I20260620 23:02:49.652653 19683 tablet_bootstrap.cc:492] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: No bootstrap required, opened a new log
I20260620 23:02:49.652755 19682 ts_tablet_manager.cc:1397] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 23:02:49.652915 19683 ts_tablet_manager.cc:1397] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260620 23:02:49.655069 19686 tablet_bootstrap.cc:654] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:49.656201 19686 log.cc:824] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:49.658528 19686 tablet_bootstrap.cc:492] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: No bootstrap required, opened a new log
I20260620 23:02:49.658774 19686 ts_tablet_manager.cc:1397] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 23:02:49.663689 19682 raft_consensus.cc:348] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.663690 19683 raft_consensus.cc:348] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.664122 19682 raft_consensus.cc:374] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:49.664119 19683 raft_consensus.cc:374] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:49.664228 19683 raft_consensus.cc:729] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e795fe13bb2640f28ba823381e23d206, State: Initialized, Role: FOLLOWER
I20260620 23:02:49.664216 19682 raft_consensus.cc:729] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1271d50966c24c0eb25f634e06944bb7, State: Initialized, Role: FOLLOWER
I20260620 23:02:49.664901 19682 consensus_queue.cc:260] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.664911 19683 consensus_queue.cc:260] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.668520 19686 raft_consensus.cc:348] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.668910 19686 raft_consensus.cc:374] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:49.669003 19686 raft_consensus.cc:729] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce75845903d6431eb2175c5ec8c70b74, State: Initialized, Role: FOLLOWER
I20260620 23:02:49.669621 19686 consensus_queue.cc:260] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.672075 19664 heartbeater.cc:503] Master 127.10.134.190:38653 was elected leader, sending a full tablet report...
I20260620 23:02:49.673367 19682 ts_tablet_manager.cc:1428] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: Time spent starting tablet: real 0.021s user 0.008s sys 0.013s
I20260620 23:02:49.674342 19403 heartbeater.cc:503] Master 127.10.134.190:38653 was elected leader, sending a full tablet report...
I20260620 23:02:49.675010 19683 ts_tablet_manager.cc:1428] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
I20260620 23:02:49.680624 19534 heartbeater.cc:503] Master 127.10.134.190:38653 was elected leader, sending a full tablet report...
I20260620 23:02:49.681625 19686 ts_tablet_manager.cc:1428] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Time spent starting tablet: real 0.023s user 0.018s sys 0.000s
I20260620 23:02:49.694257 19690 raft_consensus.cc:482] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:49.694531 19690 raft_consensus.cc:504] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.696143 19690 leader_election.cc:288] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987), e795fe13bb2640f28ba823381e23d206 (127.10.134.129:42209)
I20260620 23:02:49.703310 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "ce75845903d6431eb2175c5ec8c70b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: true
I20260620 23:02:49.703403 19619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "ce75845903d6431eb2175c5ec8c70b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1271d50966c24c0eb25f634e06944bb7" is_pre_election: true
I20260620 23:02:49.703816 19358 raft_consensus.cc:2454] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce75845903d6431eb2175c5ec8c70b74 in term 0.
I20260620 23:02:49.703890 19619 raft_consensus.cc:2454] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce75845903d6431eb2175c5ec8c70b74 in term 0.
I20260620 23:02:49.704538 19424 leader_election.cc:302] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1271d50966c24c0eb25f634e06944bb7, ce75845903d6431eb2175c5ec8c70b74, e795fe13bb2640f28ba823381e23d206; no voters:
I20260620 23:02:49.704928 19690 raft_consensus.cc:2781] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:49.705062 19690 raft_consensus.cc:482] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:49.705189 19690 raft_consensus.cc:3037] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:49.707253 19690 raft_consensus.cc:504] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.707890 19690 leader_election.cc:288] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [CANDIDATE]: Term 1 election: Requested vote from peers 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987), e795fe13bb2640f28ba823381e23d206 (127.10.134.129:42209)
I20260620 23:02:49.708230 19619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "ce75845903d6431eb2175c5ec8c70b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1271d50966c24c0eb25f634e06944bb7"
I20260620 23:02:49.708415 19619 raft_consensus.cc:3037] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:49.708411 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "ce75845903d6431eb2175c5ec8c70b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206"
I20260620 23:02:49.708617 19358 raft_consensus.cc:3037] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:49.710500 19619 raft_consensus.cc:2454] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce75845903d6431eb2175c5ec8c70b74 in term 1.
I20260620 23:02:49.710664 19358 raft_consensus.cc:2454] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce75845903d6431eb2175c5ec8c70b74 in term 1.
I20260620 23:02:49.710947 19425 leader_election.cc:302] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1271d50966c24c0eb25f634e06944bb7, ce75845903d6431eb2175c5ec8c70b74; no voters:
I20260620 23:02:49.711277 19690 raft_consensus.cc:2781] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:49.711706 19690 raft_consensus.cc:686] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [term 1 LEADER]: Becoming Leader. State: Replica: ce75845903d6431eb2175c5ec8c70b74, State: Running, Role: LEADER
I20260620 23:02:49.712206 19690 consensus_queue.cc:237] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } }
I20260620 23:02:49.716610 19215 catalog_manager.cc:5495] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce75845903d6431eb2175c5ec8c70b74 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "ce75845903d6431eb2175c5ec8c70b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce75845903d6431eb2175c5ec8c70b74" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 45365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:49.744791 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:49.746968 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e795fe13bb2640f28ba823381e23d206 to finish bootstrapping
I20260620 23:02:49.755122 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ce75845903d6431eb2175c5ec8c70b74 to finish bootstrapping
I20260620 23:02:49.763249 10778 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1271d50966c24c0eb25f634e06944bb7 to finish bootstrapping
I20260620 23:02:49.772143 19398 generic_service.cc:175] {username='slave'} at 127.0.0.1:57446 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260620 23:02:49.773820 19529 generic_service.cc:175] {username='slave'} at 127.0.0.1:58702 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260620 23:02:49.775395 19659 generic_service.cc:175] {username='slave'} at 127.0.0.1:55698 changed flags via RPC: log_inject_latency from 'false' to 'false'
W20260620 23:02:49.788187 19665 tablet.cc:2369] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:49.792933 19619 raft_consensus.cc:1261] T 4a8b1b9653ca4ed0982c13797f601c60 P 1271d50966c24c0eb25f634e06944bb7 [term 1 FOLLOWER]: Refusing update from remote peer ce75845903d6431eb2175c5ec8c70b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:49.792932 19358 raft_consensus.cc:1261] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206 [term 1 FOLLOWER]: Refusing update from remote peer ce75845903d6431eb2175c5ec8c70b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:49.793861 19693 consensus_queue.cc:1035] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1271d50966c24c0eb25f634e06944bb7" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:49.794257 19690 consensus_queue.cc:1035] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e795fe13bb2640f28ba823381e23d206" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 42209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:49.807627 19697 mvcc.cc:205] Tried to move back new op lower bound from 7299057949862916096 to 7299057949545009152. Current Snapshot: MvccSnapshot[applied={T|T < 7299057949862916096 or (T in {7299057949862916096})}]
I20260620 23:02:49.825421 19529 generic_service.cc:175] {username='slave'} at 127.0.0.1:58702 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20260620 23:02:49.826781 19529 generic_service.cc:175] {username='slave'} at 127.0.0.1:58702 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260620 23:02:49.827922 19529 generic_service.cc:175] {username='slave'} at 127.0.0.1:58702 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260620 23:02:49.829039 19398 generic_service.cc:175] {username='slave'} at 127.0.0.1:57446 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20260620 23:02:49.830057 19398 generic_service.cc:175] {username='slave'} at 127.0.0.1:57446 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260620 23:02:49.830999 19398 generic_service.cc:175] {username='slave'} at 127.0.0.1:57446 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260620 23:02:49.831413 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 19537
I20260620 23:02:49.832211 19659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:49.919999 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 19537
I20260620 23:02:49.940864 10778 test_util.cc:268] Using random seed: 1753928535
I20260620 23:02:49.954595 10778 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260620 23:02:49.967922 19694 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260620 23:02:49.968681 19425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111)
W20260620 23:02:49.969818 19701 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Injecting 600ms of latency in SegmentAllocator::Sync()
W20260620 23:02:49.970350 19425 consensus_peers.cc:487] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 -> Peer 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987): Couldn't send request to peer 1271d50966c24c0eb25f634e06944bb7. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:02:50.006361 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.009459 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.062275 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.064656 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.117478 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.120013 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.172693 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.175237 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.227705 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.230078 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.282617 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.285063 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.337872 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.340293 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.393167 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.395623 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.448063 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.450604 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.503259 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.505831 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.558622 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.561201 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.613997 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.616505 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.669193 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.672288 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.725364 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.727701 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.780423 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.782897 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.835417 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.837762 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.890452 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.892841 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:50.945511 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:50.948124 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.000756 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.003206 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.055763 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.058127 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:51.096261 19425 consensus_peers.cc:487] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 -> Peer 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987): Couldn't send request to peer 1271d50966c24c0eb25f634e06944bb7. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260620 23:02:51.110579 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.112998 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.165688 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.168294 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.170049 19715 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58722 (ReqId={client: 4a8111dd94f048589ba22871360579fd, seq_no=0, attempt_no=0}) took 1204 ms. Trace:
I20260620 23:02:51.170229 19715 rpcz_store.cc:276] 0620 23:02:49.965290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:49.965368 (+ 78us) service_pool.cc:227] Handling call
0620 23:02:51.169964 (+1204596us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:49.966748 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4a8b1b9653ca4ed0982c13797f601c60
0620 23:02:49.966889 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:49.966896 (+ 7us) write_op.cc:435] Acquired schema lock
0620 23:02:49.966899 (+ 3us) tablet.cc:660] Decoding operations
0620 23:02:49.966992 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:49.967005 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:49.967010 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:49.967063 (+ 53us) tablet.cc:699] Row locks acquired
0620 23:02:49.967065 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:49.967187 (+ 122us) write_op.cc:270] Start()
0620 23:02:49.967224 (+ 37us) write_op.cc:276] Timestamp: P: 1781996569967025 usec, L: 0
0620 23:02:49.967227 (+ 3us) op_driver.cc:331] REPLICATION: starting
0620 23:02:49.967463 (+ 236us) log.cc:842] Serialized 345 byte log entry
0620 23:02:51.168460 (+1200997us) op_driver.cc:444] REPLICATION: finished
0620 23:02:51.168950 (+ 490us) write_op.cc:301] APPLY: starting
0620 23:02:51.168972 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:51.169031 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:51.169112 (+ 81us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:51.169229 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:51.169282 (+ 53us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:51.169293 (+ 11us) write_op.cc:312] APPLY: finished
0620 23:02:51.169412 (+ 119us) log.cc:842] Serialized 22 byte log entry
0620 23:02:51.169554 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:51.169582 (+ 28us) write_op.cc:454] Released schema lock
0620 23:02:51.169593 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":394,"num_ops":1,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":862,"prepare.run_wall_time_us":862,"raft.queue_time_us":119,"raft.run_cpu_time_us":715,"raft.run_wall_time_us":715,"replication_time_us":1201208,"spinlock_wait_cycles":25984,"thread_start_us":255,"threads_started":1}]]}
I20260620 23:02:51.221500 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.223825 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.276327 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.278741 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.331475 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.333860 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:51.375470 19694 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260620 23:02:51.377230 19701 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260620 23:02:51.386575 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.389040 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.441789 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.444247 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.496918 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.499562 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.552201 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.554636 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.607316 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.609781 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.662693 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.666047 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.719283 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.721635 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.774457 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.776857 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.829403 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.831879 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.884644 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.887221 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.939795 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.942248 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:51.994735 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:51.997246 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.049885 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.052313 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:52.093458 19425 consensus_peers.cc:487] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 -> Peer 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987): Couldn't send request to peer 1271d50966c24c0eb25f634e06944bb7. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260620 23:02:52.104801 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.107287 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.159868 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.162300 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.215054 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.217751 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.270417 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.272820 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.325429 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.327906 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.380519 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.383119 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.435971 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.438596 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.491048 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.493337 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.546068 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.548635 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.577256 19721 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58722 (ReqId={client: 4a8111dd94f048589ba22871360579fd, seq_no=1, attempt_no=0}) took 1203 ms. Trace:
I20260620 23:02:52.577538 19721 rpcz_store.cc:276] 0620 23:02:51.373342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:51.373420 (+ 78us) service_pool.cc:227] Handling call
0620 23:02:52.577178 (+1203758us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:51.374465 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4a8b1b9653ca4ed0982c13797f601c60
0620 23:02:51.374576 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:51.374581 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:51.374582 (+ 1us) tablet.cc:660] Decoding operations
0620 23:02:51.374698 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:51.374709 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:51.374712 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:51.374763 (+ 51us) tablet.cc:699] Row locks acquired
0620 23:02:51.374765 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:51.374803 (+ 38us) write_op.cc:270] Start()
0620 23:02:51.374839 (+ 36us) write_op.cc:276] Timestamp: P: 1781996571374689 usec, L: 0
0620 23:02:51.374842 (+ 3us) op_driver.cc:331] REPLICATION: starting
0620 23:02:51.375048 (+ 206us) log.cc:842] Serialized 354 byte log entry
0620 23:02:52.576104 (+1201056us) op_driver.cc:444] REPLICATION: finished
0620 23:02:52.576566 (+ 462us) write_op.cc:301] APPLY: starting
0620 23:02:52.576584 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:52.576622 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:52.576655 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:52.576730 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:52.576756 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:52.576763 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:02:52.576845 (+ 82us) log.cc:842] Serialized 22 byte log entry
0620 23:02:52.577033 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:52.577057 (+ 24us) write_op.cc:454] Released schema lock
0620 23:02:52.577065 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":384,"num_ops":1,"prepare.queue_time_us":387,"prepare.run_cpu_time_us":736,"prepare.run_wall_time_us":737,"raft.queue_time_us":167,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":330,"replication_time_us":1201231,"thread_start_us":474,"threads_started":2}]]}
I20260620 23:02:52.601285 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.603713 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.656275 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.658911 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.711514 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.714036 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.766738 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.769208 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:52.781373 19694 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260620 23:02:52.783115 19701 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260620 23:02:52.822372 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.825116 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.877714 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.880084 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.932752 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.935431 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:52.987999 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:52.990592 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:53.021164 19425 consensus_peers.cc:487] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 -> Peer 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987): Couldn't send request to peer 1271d50966c24c0eb25f634e06944bb7. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260620 23:02:53.043102 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.045408 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.098088 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.100476 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.153244 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.155755 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.208461 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.210822 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.263500 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.266098 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.318830 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.321135 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.373888 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.376371 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.429184 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.431625 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.484261 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.486701 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.539369 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.542061 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.594642 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.597023 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.649716 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.652909 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.705829 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.708290 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.760895 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.764415 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.817492 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.819983 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.873159 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.875573 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.928294 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.930703 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:53.983175 19727 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58722 (ReqId={client: 4a8111dd94f048589ba22871360579fd, seq_no=2, attempt_no=0}) took 1203 ms. Trace:
I20260620 23:02:53.983601 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:53.983425 19727 rpcz_store.cc:276] 0620 23:02:52.779687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:52.779767 (+ 80us) service_pool.cc:227] Handling call
0620 23:02:53.983099 (+1203332us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:52.780556 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4a8b1b9653ca4ed0982c13797f601c60
0620 23:02:52.780653 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:52.780657 (+ 4us) write_op.cc:435] Acquired schema lock
0620 23:02:52.780659 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:52.780743 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:52.780754 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:52.780756 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:52.780803 (+ 47us) tablet.cc:699] Row locks acquired
0620 23:02:52.780805 (+ 2us) write_op.cc:260] PREPARE: finished
0620 23:02:52.780838 (+ 33us) write_op.cc:270] Start()
0620 23:02:52.780875 (+ 37us) write_op.cc:276] Timestamp: P: 1781996572780767 usec, L: 0
0620 23:02:52.780877 (+ 2us) op_driver.cc:331] REPLICATION: starting
0620 23:02:52.781045 (+ 168us) log.cc:842] Serialized 354 byte log entry
0620 23:02:53.981927 (+1200882us) op_driver.cc:444] REPLICATION: finished
0620 23:02:53.982407 (+ 480us) write_op.cc:301] APPLY: starting
0620 23:02:53.982450 (+ 43us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:53.982496 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:53.982593 (+ 97us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:53.982691 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:53.982731 (+ 40us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:53.982738 (+ 7us) write_op.cc:312] APPLY: finished
0620 23:02:53.982845 (+ 107us) log.cc:842] Serialized 22 byte log entry
0620 23:02:53.982962 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:53.982985 (+ 23us) write_op.cc:454] Released schema lock
0620 23:02:53.982994 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":392,"num_ops":1,"prepare.queue_time_us":334,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":651,"raft.queue_time_us":139,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":280,"replication_time_us":1201019,"thread_start_us":486,"threads_started":2}]]}
I20260620 23:02:53.987051 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:54.001104 19425 consensus_peers.cc:487] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 -> Peer 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987): Couldn't send request to peer 1271d50966c24c0eb25f634e06944bb7. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260620 23:02:54.039697 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.042387 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.094889 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.097265 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.149885 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.152478 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:54.188318 19694 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260620 23:02:54.190027 19701 log.cc:508] T 4a8b1b9653ca4ed0982c13797f601c60 P e795fe13bb2640f28ba823381e23d206: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260620 23:02:54.204965 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.207388 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.259997 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.262560 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.315261 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.317737 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.370406 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.373104 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.425971 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.428334 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.481210 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.483906 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.536751 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.540081 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.593544 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.595922 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.648489 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.650753 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.703352 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.705688 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.758550 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.761178 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.813839 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.816439 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.869028 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.871536 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.924119 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.926649 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:54.979359 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:54.981869 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.034570 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.037002 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
W20260620 23:02:55.062772 19425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20260620 23:02:55.063747 19425 consensus_peers.cc:487] T 4a8b1b9653ca4ed0982c13797f601c60 P ce75845903d6431eb2175c5ec8c70b74 -> Peer 1271d50966c24c0eb25f634e06944bb7 (127.10.134.131:42987): Couldn't send request to peer 1271d50966c24c0eb25f634e06944bb7. Status: Network error: Client connection negotiation failed: client connection to 127.10.134.131:42987: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260620 23:02:55.089576 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.092037 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.145116 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.147723 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.200261 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.202889 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.255438 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.257926 19358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.310639 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.313318 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.365993 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.368309 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.390021 19733 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58722 (ReqId={client: 4a8111dd94f048589ba22871360579fd, seq_no=3, attempt_no=0}) took 1203 ms. Trace:
I20260620 23:02:55.390281 19733 rpcz_store.cc:276] 0620 23:02:54.186438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0620 23:02:54.186519 (+ 81us) service_pool.cc:227] Handling call
0620 23:02:55.389898 (+1203379us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0620 23:02:54.187448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4a8b1b9653ca4ed0982c13797f601c60
0620 23:02:54.187546 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0620 23:02:54.187551 (+ 5us) write_op.cc:435] Acquired schema lock
0620 23:02:54.187553 (+ 2us) tablet.cc:660] Decoding operations
0620 23:02:54.187630 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0620 23:02:54.187641 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0620 23:02:54.187644 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0620 23:02:54.187703 (+ 59us) tablet.cc:699] Row locks acquired
0620 23:02:54.187704 (+ 1us) write_op.cc:260] PREPARE: finished
0620 23:02:54.187741 (+ 37us) write_op.cc:270] Start()
0620 23:02:54.187775 (+ 34us) write_op.cc:276] Timestamp: P: 1781996574187700 usec, L: 0
0620 23:02:54.187778 (+ 3us) op_driver.cc:331] REPLICATION: starting
0620 23:02:54.187964 (+ 186us) log.cc:842] Serialized 354 byte log entry
0620 23:02:55.388833 (+1200869us) op_driver.cc:444] REPLICATION: finished
0620 23:02:55.389310 (+ 477us) write_op.cc:301] APPLY: starting
0620 23:02:55.389328 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0620 23:02:55.389364 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0620 23:02:55.389394 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0620 23:02:55.389474 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0620 23:02:55.389501 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0620 23:02:55.389505 (+ 4us) write_op.cc:312] APPLY: finished
0620 23:02:55.389585 (+ 80us) log.cc:842] Serialized 22 byte log entry
0620 23:02:55.389741 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0620 23:02:55.389766 (+ 25us) write_op.cc:454] Released schema lock
0620 23:02:55.389774 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":411,"mutex_wait_us":25,"num_ops":1,"prepare.queue_time_us":424,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":717,"raft.queue_time_us":160,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":296,"replication_time_us":1201026,"spinlock_wait_cycles":6528,"thread_start_us":561,"threads_started":2}]]}
I20260620 23:02:55.420804 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.423247 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.475865 19489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "ce75845903d6431eb2175c5ec8c70b74" is_pre_election: false
I20260620 23:02:55.478425 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b1b9653ca4ed0982c13797f601c60" candidate_uuid: "1271d50966c24c0eb25f634e06944bb7" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "e795fe13bb2640f28ba823381e23d206" is_pre_election: false
I20260620 23:02:55.594151 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 19276
I20260620 23:02:55.595240 19398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:55.682698 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 19276
I20260620 23:02:55.700196 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 19407
I20260620 23:02:55.701120 19529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:55.793155 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 19407
I20260620 23:02:55.810412 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 19185
I20260620 23:02:55.811601 19246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:55.917359 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 19185
2026-06-20T23:02:55Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (8275 ms)
[----------] 12 tests from RaftConsensusElectionITest (194609 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20260620 23:02:55.957046 10778 test_util.cc:268] Using random seed: 1759944715
I20260620 23:02:55.958829 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:55.958930 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:55.958983 10778 ts_itest-base.cc:114] 1 tablet servers
I20260620 23:02:55.959048 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:02:55.959100 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:55Z Disabled control of system clock
I20260620 23:02:55.997653 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:36651
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:36651
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260620 23:02:56.319391 19747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:56.319669 19747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:56.319752 19747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:56.326730 19747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:56.326826 19747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:56.326867 19747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:56.326921 19747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:56.335705 19747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:36651
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:36651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:56.336925 19747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:56.338701 19747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:56.350621 19752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:56.350751 19753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:56.350790 19747 server_base.cc:1034] running on GCE node
W20260620 23:02:56.350661 19755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:56.351936 19747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:56.353267 19747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:56.354472 19747 hybrid_clock.cc:648] HybridClock initialized: now 1781996576354416 us; error 34 us; skew 500 ppm
I20260620 23:02:56.354799 19747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:56.357074 19747 webserver.cc:460] Webserver started at http://127.10.134.190:35547/ using document root <none> and password file <none>
I20260620 23:02:56.357740 19747 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:56.357885 19747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:56.358209 19747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:56.362095 19747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2d58486bfb7e4b25a157ffd5d3a32c40"
format_stamp: "Formatted at 2026-06-20 23:02:56 on dist-test-slave-8k25"
I20260620 23:02:56.362824 19747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2d58486bfb7e4b25a157ffd5d3a32c40"
format_stamp: "Formatted at 2026-06-20 23:02:56 on dist-test-slave-8k25"
I20260620 23:02:56.367672 19747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:02:56.370873 19761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:56.372164 19747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:56.372433 19747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2d58486bfb7e4b25a157ffd5d3a32c40"
format_stamp: "Formatted at 2026-06-20 23:02:56 on dist-test-slave-8k25"
I20260620 23:02:56.372622 19747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:56.414176 19747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:56.415028 19747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:56.415351 19747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:56.433317 19747 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:36651
I20260620 23:02:56.433351 19812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:36651 every 8 connection(s)
I20260620 23:02:56.434684 19747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:56.438393 19813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:56.441535 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 19747
I20260620 23:02:56.441754 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:56.450795 19813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40: Bootstrap starting.
I20260620 23:02:56.454355 19813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:56.455502 19813 log.cc:824] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:56.458141 19813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40: No bootstrap required, opened a new log
I20260620 23:02:56.464836 19813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 36651 } }
I20260620 23:02:56.465184 19813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:56.465276 19813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d58486bfb7e4b25a157ffd5d3a32c40, State: Initialized, Role: FOLLOWER
I20260620 23:02:56.465736 19813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 36651 } }
I20260620 23:02:56.465922 19813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:56.466085 19813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:56.466202 19813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:56.468412 19813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 36651 } }
I20260620 23:02:56.468816 19813 leader_election.cc:302] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d58486bfb7e4b25a157ffd5d3a32c40; no voters:
I20260620 23:02:56.469343 19813 leader_election.cc:288] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:56.469511 19818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:56.470261 19818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [term 1 LEADER]: Becoming Leader. State: Replica: 2d58486bfb7e4b25a157ffd5d3a32c40, State: Running, Role: LEADER
I20260620 23:02:56.470733 19818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 36651 } }
I20260620 23:02:56.471338 19813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:56.474218 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d58486bfb7e4b25a157ffd5d3a32c40. Latest consensus state: current_term: 1 leader_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 36651 } } }
I20260620 23:02:56.474659 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:56.478092 19819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d58486bfb7e4b25a157ffd5d3a32c40" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 36651 } } }
I20260620 23:02:56.478546 19819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:56.478964 19827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:56.483978 19827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:56.491768 19827 catalog_manager.cc:1347] Generated new cluster ID: b95cb5fd57894d58b4d5fd72694de405
I20260620 23:02:56.491888 19827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:56.509078 19827 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:56.510438 19827 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:56.519183 19827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d58486bfb7e4b25a157ffd5d3a32c40: Generated new TSK 0
I20260620 23:02:56.520232 19827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:56.526837 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:36651
--builtin_ntp_servers=127.10.134.148:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260620 23:02:56.853824 19837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:56.854274 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:56.854382 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:56.854512 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:56.861610 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:56.861825 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:56.870512 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:36651
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:56.871742 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:56.873519 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:56.886828 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:56.886828 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:56.886863 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:56.887498 19837 server_base.cc:1034] running on GCE node
I20260620 23:02:56.888103 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:56.888981 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:56.890203 19837 hybrid_clock.cc:648] HybridClock initialized: now 1781996576890158 us; error 55 us; skew 500 ppm
I20260620 23:02:56.890516 19837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:56.892736 19837 webserver.cc:460] Webserver started at http://127.10.134.129:34679/ using document root <none> and password file <none>
I20260620 23:02:56.893429 19837 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:56.893558 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:56.893855 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:56.896322 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "01aefc863f874e81b04a18652c8bf44b"
format_stamp: "Formatted at 2026-06-20 23:02:56 on dist-test-slave-8k25"
I20260620 23:02:56.897042 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "01aefc863f874e81b04a18652c8bf44b"
format_stamp: "Formatted at 2026-06-20 23:02:56 on dist-test-slave-8k25"
I20260620 23:02:56.901996 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:02:56.905262 19851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:56.906641 19837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 23:02:56.906813 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "01aefc863f874e81b04a18652c8bf44b"
format_stamp: "Formatted at 2026-06-20 23:02:56 on dist-test-slave-8k25"
I20260620 23:02:56.907025 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:56.919832 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:56.920598 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:56.920902 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:56.921840 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:56.923205 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:56.923323 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:56.923465 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:56.923552 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:56.955735 19837 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:38565
I20260620 23:02:56.955768 19963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:38565 every 8 connection(s)
I20260620 23:02:56.957082 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:56.964493 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 19837
I20260620 23:02:56.964684 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:56.970574 19964 heartbeater.cc:347] Connected to a master server at 127.10.134.190:36651
I20260620 23:02:56.970861 19964 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:56.971591 19964 heartbeater.cc:511] Master 127.10.134.190:36651 requested a full tablet report, sending...
I20260620 23:02:56.973580 19778 ts_manager.cc:194] Registered new tserver with Master: 01aefc863f874e81b04a18652c8bf44b (127.10.134.129:38565)
I20260620 23:02:56.975069 19778 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:49785
I20260620 23:02:56.980072 10778 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260620 23:02:56.994691 10778 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260620 23:02:56.994800 10778 test_util.cc:268] Using random seed: 1760982479
I20260620 23:02:57.013535 19778 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50832:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:02:57.041188 19897 tablet_service.cc:1467] Processing CreateTablet for tablet eff9622394014f1fab889ca28d67df49 (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 23:02:57.041069 19890 tablet_service.cc:1467] Processing CreateTablet for tablet 3a89ffd234024fe9ab92972d6b589e3f (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 23:02:57.040994 19898 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9016097f2446128fcdc5a3d64e67c6 (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 23:02:57.042273 19896 tablet_service.cc:1467] Processing CreateTablet for tablet 61b4cb3045e3497b91026002a77d500c (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 23:02:57.042685 19891 tablet_service.cc:1467] Processing CreateTablet for tablet d49e4655b26443b28d87fb93aa906be0 (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 23:02:57.043154 19897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff9622394014f1fab889ca28d67df49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.043201 19893 tablet_service.cc:1467] Processing CreateTablet for tablet 6db52abd244340869c80c6a9ac63dd39 (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 23:02:57.043627 19893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db52abd244340869c80c6a9ac63dd39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.043828 19896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b4cb3045e3497b91026002a77d500c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.044066 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d49e4655b26443b28d87fb93aa906be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.044364 19898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e9016097f2446128fcdc5a3d64e67c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.041026 19892 tablet_service.cc:1467] Processing CreateTablet for tablet 34116438680b4d339abf3dfabbb0f3b4 (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 23:02:57.044847 19892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34116438680b4d339abf3dfabbb0f3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.045130 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a89ffd234024fe9ab92972d6b589e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.045306 19899 tablet_service.cc:1467] Processing CreateTablet for tablet 8caa65f001d84175bd84bade7b7c2efc (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 23:02:57.045743 19899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8caa65f001d84175bd84bade7b7c2efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.041126 19894 tablet_service.cc:1467] Processing CreateTablet for tablet 7dea3db74e144c50a10ff25e002e2e10 (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 23:02:57.046473 19894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dea3db74e144c50a10ff25e002e2e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.041675 19895 tablet_service.cc:1467] Processing CreateTablet for tablet 943c27eb7d794c4db44fe51af5dd7cdd (DEFAULT_TABLE table=test-workload [id=f6a32dc52cd940e898dbd21c17515132]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 23:02:57.047211 19895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943c27eb7d794c4db44fe51af5dd7cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:57.058743 19982 tablet_bootstrap.cc:492] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.062983 19982 tablet_bootstrap.cc:654] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.064356 19982 log.cc:824] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:57.073012 19982 tablet_bootstrap.cc:492] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.073266 19982 ts_tablet_manager.cc:1397] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260620 23:02:57.080439 19982 raft_consensus.cc:348] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.080819 19982 raft_consensus.cc:729] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.081229 19982 consensus_queue.cc:260] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.082244 19964 heartbeater.cc:503] Master 127.10.134.190:36651 was elected leader, sending a full tablet report...
I20260620 23:02:57.082743 19982 ts_tablet_manager.cc:1428] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.009s user 0.009s sys 0.002s
I20260620 23:02:57.083240 19982 tablet_bootstrap.cc:492] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.084920 19982 tablet_bootstrap.cc:654] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.086620 19982 tablet_bootstrap.cc:492] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.086766 19982 ts_tablet_manager.cc:1397] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:02:57.087462 19982 raft_consensus.cc:348] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.087641 19982 raft_consensus.cc:729] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.087832 19982 consensus_queue.cc:260] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.088274 19982 ts_tablet_manager.cc:1428] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 23:02:57.088526 19982 tablet_bootstrap.cc:492] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.089706 19982 tablet_bootstrap.cc:654] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.091683 19982 tablet_bootstrap.cc:492] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.091827 19982 ts_tablet_manager.cc:1397] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:57.092505 19982 raft_consensus.cc:348] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.092703 19982 raft_consensus.cc:729] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.092871 19982 consensus_queue.cc:260] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.093364 19982 ts_tablet_manager.cc:1428] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:57.093614 19982 tablet_bootstrap.cc:492] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.095001 19982 tablet_bootstrap.cc:654] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.096709 19982 tablet_bootstrap.cc:492] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.096851 19982 ts_tablet_manager.cc:1397] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:57.097506 19982 raft_consensus.cc:348] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.097658 19982 raft_consensus.cc:729] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.097841 19982 consensus_queue.cc:260] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.098297 19982 ts_tablet_manager.cc:1428] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:57.098548 19982 tablet_bootstrap.cc:492] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.099633 19982 tablet_bootstrap.cc:654] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.101197 19982 tablet_bootstrap.cc:492] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.101351 19982 ts_tablet_manager.cc:1397] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:57.102072 19982 raft_consensus.cc:348] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.102237 19982 raft_consensus.cc:729] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.102422 19982 consensus_queue.cc:260] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.102913 19982 ts_tablet_manager.cc:1428] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:57.103158 19982 tablet_bootstrap.cc:492] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.104665 19982 tablet_bootstrap.cc:654] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.106345 19982 tablet_bootstrap.cc:492] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.106511 19982 ts_tablet_manager.cc:1397] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:57.107069 19982 raft_consensus.cc:348] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.107249 19982 raft_consensus.cc:729] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.107445 19982 consensus_queue.cc:260] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.107858 19982 ts_tablet_manager.cc:1428] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:02:57.108080 19982 tablet_bootstrap.cc:492] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.109187 19982 tablet_bootstrap.cc:654] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.110864 19982 tablet_bootstrap.cc:492] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.111038 19982 ts_tablet_manager.cc:1397] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:02:57.111699 19982 raft_consensus.cc:348] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.111850 19982 raft_consensus.cc:729] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.111984 19982 consensus_queue.cc:260] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.112391 19982 ts_tablet_manager.cc:1428] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:02:57.112632 19982 tablet_bootstrap.cc:492] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.113719 19982 tablet_bootstrap.cc:654] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.115518 19982 tablet_bootstrap.cc:492] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.115685 19982 ts_tablet_manager.cc:1397] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:57.116381 19982 raft_consensus.cc:348] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.116602 19982 raft_consensus.cc:729] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.116768 19982 consensus_queue.cc:260] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.117154 19982 ts_tablet_manager.cc:1428] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:02:57.117396 19982 tablet_bootstrap.cc:492] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.118827 19982 tablet_bootstrap.cc:654] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.120481 19982 tablet_bootstrap.cc:492] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.120637 19982 ts_tablet_manager.cc:1397] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:02:57.121201 19982 raft_consensus.cc:348] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.121379 19982 raft_consensus.cc:729] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.121573 19982 consensus_queue.cc:260] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.122053 19982 ts_tablet_manager.cc:1428] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:57.122313 19982 tablet_bootstrap.cc:492] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: Bootstrap starting.
I20260620 23:02:57.123478 19982 tablet_bootstrap.cc:654] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:57.125243 19982 tablet_bootstrap.cc:492] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: No bootstrap required, opened a new log
I20260620 23:02:57.125403 19982 ts_tablet_manager.cc:1397] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:02:57.126080 19982 raft_consensus.cc:348] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.126255 19982 raft_consensus.cc:729] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Initialized, Role: FOLLOWER
I20260620 23:02:57.126437 19982 consensus_queue.cc:260] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.126917 19982 ts_tablet_manager.cc:1428] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:57.162356 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "943c27eb7d794c4db44fe51af5dd7cdd"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.162710 19919 raft_consensus.cc:482] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.162899 19919 raft_consensus.cc:3037] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.164909 19919 raft_consensus.cc:504] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.165469 19919 leader_election.cc:302] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.165795 19919 leader_election.cc:288] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.165881 19984 raft_consensus.cc:2781] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.166646 19984 raft_consensus.cc:686] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.167212 19984 consensus_queue.cc:237] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.171703 19778 catalog_manager.cc:5495] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.180579 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8caa65f001d84175bd84bade7b7c2efc"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.180787 19919 raft_consensus.cc:482] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.180929 19919 raft_consensus.cc:3037] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.182852 19919 raft_consensus.cc:504] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.183063 19919 leader_election.cc:302] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.183210 19919 leader_election.cc:288] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.183315 19984 raft_consensus.cc:2781] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.183534 19984 raft_consensus.cc:686] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.183761 19984 consensus_queue.cc:237] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.186384 19778 catalog_manager.cc:5495] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.194715 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff9622394014f1fab889ca28d67df49"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.194886 19919 raft_consensus.cc:482] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.195050 19919 raft_consensus.cc:3037] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.196863 19919 raft_consensus.cc:504] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.197062 19919 leader_election.cc:302] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.197214 19919 leader_election.cc:288] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.197327 19984 raft_consensus.cc:2781] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.197517 19984 raft_consensus.cc:686] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.197747 19984 consensus_queue.cc:237] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.200106 19778 catalog_manager.cc:5495] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.208600 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61b4cb3045e3497b91026002a77d500c"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.208809 19919 raft_consensus.cc:482] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
W20260620 23:02:57.208833 19965 tablet.cc:2369] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:57.208961 19919 raft_consensus.cc:3037] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.211067 19919 raft_consensus.cc:504] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.211308 19919 leader_election.cc:302] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.211506 19919 leader_election.cc:288] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.211601 19984 raft_consensus.cc:2781] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.211796 19984 raft_consensus.cc:686] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.212030 19984 consensus_queue.cc:237] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.214406 19778 catalog_manager.cc:5495] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.223081 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34116438680b4d339abf3dfabbb0f3b4"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.223289 19919 raft_consensus.cc:482] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.223405 19919 raft_consensus.cc:3037] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.225190 19919 raft_consensus.cc:504] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.225430 19919 leader_election.cc:302] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.225620 19919 leader_election.cc:288] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.225730 19984 raft_consensus.cc:2781] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.225921 19984 raft_consensus.cc:686] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.226184 19984 consensus_queue.cc:237] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.228546 19778 catalog_manager.cc:5495] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.236868 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a89ffd234024fe9ab92972d6b589e3f"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.237020 19919 raft_consensus.cc:482] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.237110 19919 raft_consensus.cc:3037] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.238893 19919 raft_consensus.cc:504] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.239082 19919 leader_election.cc:302] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.239243 19919 leader_election.cc:288] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.239300 19984 raft_consensus.cc:2781] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.239457 19984 raft_consensus.cc:686] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.239677 19984 consensus_queue.cc:237] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.241851 19778 catalog_manager.cc:5495] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.250388 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7dea3db74e144c50a10ff25e002e2e10"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.250586 19919 raft_consensus.cc:482] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.250720 19919 raft_consensus.cc:3037] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.252569 19919 raft_consensus.cc:504] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.252806 19919 leader_election.cc:302] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.252998 19919 leader_election.cc:288] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.253115 19984 raft_consensus.cc:2781] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.253300 19984 raft_consensus.cc:686] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.253551 19984 consensus_queue.cc:237] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.255756 19778 catalog_manager.cc:5495] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.264684 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6db52abd244340869c80c6a9ac63dd39"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.264860 19919 raft_consensus.cc:482] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.264972 19919 raft_consensus.cc:3037] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.266793 19919 raft_consensus.cc:504] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.267016 19919 leader_election.cc:302] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.267197 19919 leader_election.cc:288] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.267266 19984 raft_consensus.cc:2781] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.267493 19984 raft_consensus.cc:686] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.267729 19984 consensus_queue.cc:237] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.269969 19778 catalog_manager.cc:5495] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.278945 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d49e4655b26443b28d87fb93aa906be0"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.279161 19919 raft_consensus.cc:482] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.279297 19919 raft_consensus.cc:3037] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.281024 19919 raft_consensus.cc:504] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.281296 19919 leader_election.cc:302] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.281518 19919 leader_election.cc:288] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.281594 19984 raft_consensus.cc:2781] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.281737 19984 raft_consensus.cc:686] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.282271 19984 consensus_queue.cc:237] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.284562 19778 catalog_manager.cc:5495] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.292789 19919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e9016097f2446128fcdc5a3d64e67c6"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.292979 19919 raft_consensus.cc:482] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:02:57.293115 19919 raft_consensus.cc:3037] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:57.294956 19919 raft_consensus.cc:504] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.295202 19919 leader_election.cc:302] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01aefc863f874e81b04a18652c8bf44b; no voters:
I20260620 23:02:57.295418 19919 leader_election.cc:288] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:57.295528 19984 raft_consensus.cc:2781] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:57.295735 19984 raft_consensus.cc:686] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Leader. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.295995 19984 consensus_queue.cc:237] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.298305 19778 catalog_manager.cc:5495] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01aefc863f874e81b04a18652c8bf44b (127.10.134.129). New cstate: current_term: 1 leader_uuid: "01aefc863f874e81b04a18652c8bf44b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:57.307277 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 943c27eb7d794c4db44fe51af5dd7cdd with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.308092 20008 tablet_replica.cc:330] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.308527 20008 raft_consensus.cc:2229] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:57.308892 20008 raft_consensus.cc:2258] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.310770 20008 ts_tablet_manager.cc:1905] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.315274 20008 ts_tablet_manager.cc:1918] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.315433 20008 log.cc:1196] T 943c27eb7d794c4db44fe51af5dd7cdd P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/943c27eb7d794c4db44fe51af5dd7cdd
I20260620 23:02:57.325117 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8caa65f001d84175bd84bade7b7c2efc with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.325420 20008 tablet_replica.cc:330] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.325636 20008 raft_consensus.cc:2229] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:57.325842 20008 raft_consensus.cc:2258] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.326721 20008 ts_tablet_manager.cc:1905] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.330749 20008 ts_tablet_manager.cc:1918] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.330878 20008 log.cc:1196] T 8caa65f001d84175bd84bade7b7c2efc P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/8caa65f001d84175bd84bade7b7c2efc
I20260620 23:02:57.340327 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet eff9622394014f1fab889ca28d67df49 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.340626 20008 tablet_replica.cc:330] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.340827 20008 raft_consensus.cc:2229] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:57.341040 20008 raft_consensus.cc:2258] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.341821 20008 ts_tablet_manager.cc:1905] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.345698 20008 ts_tablet_manager.cc:1918] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.345796 20008 log.cc:1196] T eff9622394014f1fab889ca28d67df49 P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/eff9622394014f1fab889ca28d67df49
I20260620 23:02:57.355386 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b4cb3045e3497b91026002a77d500c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.355691 20008 tablet_replica.cc:330] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.355887 20008 raft_consensus.cc:2229] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:57.356146 20008 raft_consensus.cc:2258] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.356995 20008 ts_tablet_manager.cc:1905] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.360961 20008 ts_tablet_manager.cc:1918] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.361091 20008 log.cc:1196] T 61b4cb3045e3497b91026002a77d500c P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/61b4cb3045e3497b91026002a77d500c
I20260620 23:02:57.370311 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 34116438680b4d339abf3dfabbb0f3b4 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.370586 20008 tablet_replica.cc:330] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.370779 20008 raft_consensus.cc:2229] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:57.371043 20008 raft_consensus.cc:2258] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.371831 20008 ts_tablet_manager.cc:1905] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.375926 20008 ts_tablet_manager.cc:1918] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.376036 20008 log.cc:1196] T 34116438680b4d339abf3dfabbb0f3b4 P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/34116438680b4d339abf3dfabbb0f3b4
I20260620 23:02:57.385756 19919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a89ffd234024fe9ab92972d6b589e3f"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.386049 19919 raft_consensus.cc:580] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Received request to step down
I20260620 23:02:57.386185 19919 raft_consensus.cc:3032] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:57.386279 19919 raft_consensus.cc:729] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.386570 19919 consensus_queue.cc:260] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.386760 19919 raft_consensus.cc:3037] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:57.396134 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a89ffd234024fe9ab92972d6b589e3f with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.396469 20008 tablet_replica.cc:330] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.396625 20008 raft_consensus.cc:2229] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:57.396790 20008 raft_consensus.cc:2258] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.397670 20008 ts_tablet_manager.cc:1905] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.401837 20008 ts_tablet_manager.cc:1918] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.402019 20008 log.cc:1196] T 3a89ffd234024fe9ab92972d6b589e3f P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a89ffd234024fe9ab92972d6b589e3f
I20260620 23:02:57.412665 19919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dea3db74e144c50a10ff25e002e2e10"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.412875 19919 raft_consensus.cc:580] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Received request to step down
I20260620 23:02:57.413015 19919 raft_consensus.cc:3032] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:57.413089 19919 raft_consensus.cc:729] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.413333 19919 consensus_queue.cc:260] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.413463 19919 raft_consensus.cc:3037] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:57.422634 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dea3db74e144c50a10ff25e002e2e10 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.422977 20008 tablet_replica.cc:330] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.423167 20008 raft_consensus.cc:2229] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:57.423362 20008 raft_consensus.cc:2258] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.424232 20008 ts_tablet_manager.cc:1905] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.428216 20008 ts_tablet_manager.cc:1918] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.428370 20008 log.cc:1196] T 7dea3db74e144c50a10ff25e002e2e10 P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/7dea3db74e144c50a10ff25e002e2e10
I20260620 23:02:57.441242 19919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6db52abd244340869c80c6a9ac63dd39"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.441454 19919 raft_consensus.cc:580] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Received request to step down
I20260620 23:02:57.441560 19919 raft_consensus.cc:3032] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:57.441632 19919 raft_consensus.cc:729] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.441862 19919 consensus_queue.cc:260] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.442086 19919 raft_consensus.cc:3037] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:57.457202 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db52abd244340869c80c6a9ac63dd39 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.457504 20008 tablet_replica.cc:330] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.457681 20008 raft_consensus.cc:2229] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:57.457903 20008 raft_consensus.cc:2258] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.458843 20008 ts_tablet_manager.cc:1905] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.462961 20008 ts_tablet_manager.cc:1918] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.463058 20008 log.cc:1196] T 6db52abd244340869c80c6a9ac63dd39 P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/6db52abd244340869c80c6a9ac63dd39
I20260620 23:02:57.472474 19919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d49e4655b26443b28d87fb93aa906be0"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.472677 19919 raft_consensus.cc:580] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Received request to step down
I20260620 23:02:57.472774 19919 raft_consensus.cc:3032] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:57.472823 19919 raft_consensus.cc:729] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.472988 19919 consensus_queue.cc:260] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.473109 19919 raft_consensus.cc:3037] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:57.482035 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet d49e4655b26443b28d87fb93aa906be0 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.482324 20008 tablet_replica.cc:330] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.482455 20008 raft_consensus.cc:2229] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:57.482626 20008 raft_consensus.cc:2258] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.483449 20008 ts_tablet_manager.cc:1905] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.487362 20008 ts_tablet_manager.cc:1918] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.487460 20008 log.cc:1196] T d49e4655b26443b28d87fb93aa906be0 P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/d49e4655b26443b28d87fb93aa906be0
I20260620 23:02:57.497097 19919 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e9016097f2446128fcdc5a3d64e67c6"
dest_uuid: "01aefc863f874e81b04a18652c8bf44b"
from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.497306 19919 raft_consensus.cc:580] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Received request to step down
I20260620 23:02:57.497421 19919 raft_consensus.cc:3032] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:02:57.497509 19919 raft_consensus.cc:729] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 01aefc863f874e81b04a18652c8bf44b, State: Running, Role: LEADER
I20260620 23:02:57.497735 19919 consensus_queue.cc:260] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01aefc863f874e81b04a18652c8bf44b" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38565 } }
I20260620 23:02:57.497889 19919 raft_consensus.cc:3037] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:57.506858 19895 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9016097f2446128fcdc5a3d64e67c6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:36358
I20260620 23:02:57.507149 20008 tablet_replica.cc:330] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: stopping tablet replica
I20260620 23:02:57.507282 20008 raft_consensus.cc:2229] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:57.507433 20008 raft_consensus.cc:2258] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:57.508237 20008 ts_tablet_manager.cc:1905] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:02:57.512302 20008 ts_tablet_manager.cc:1918] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:02:57.512398 20008 log.cc:1196] T 0e9016097f2446128fcdc5a3d64e67c6 P 01aefc863f874e81b04a18652c8bf44b: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e9016097f2446128fcdc5a3d64e67c6
I20260620 23:02:57.523507 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 19837
I20260620 23:02:57.530170 19959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:57.638970 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 19837
I20260620 23:02:57.658880 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 19747
I20260620 23:02:57.660097 19808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:02:57.763572 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 19747
2026-06-20T23:02:57Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (1844 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20260620 23:02:57.801307 10778 test_util.cc:268] Using random seed: 1761788996
I20260620 23:02:57.803015 10778 ts_itest-base.cc:112] Starting cluster with:
I20260620 23:02:57.803072 10778 ts_itest-base.cc:113] --------------
I20260620 23:02:57.803145 10778 ts_itest-base.cc:114] 3 tablet servers
I20260620 23:02:57.803193 10778 ts_itest-base.cc:115] 3 replicas per TS
I20260620 23:02:57.803260 10778 ts_itest-base.cc:116] --------------
2026-06-20T23:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:57Z Disabled control of system clock
I20260620 23:02:57.840924 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:44973
--webserver_interface=127.10.134.190
--webserver_port=0
--builtin_ntp_servers=127.10.134.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.134.190:44973
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260620 23:02:58.159384 20019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:58.159657 20019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:58.159835 20019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:58.167035 20019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:58.167133 20019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:58.167186 20019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:58.167277 20019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:58.176232 20019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.10.134.190:44973
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.134.190:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.10.134.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:58.177479 20019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:58.179209 20019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:58.190745 20024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:58.190686 20025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:58.191118 20027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:58.191802 20019 server_base.cc:1034] running on GCE node
I20260620 23:02:58.192616 20019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:58.194013 20019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:58.195271 20019 hybrid_clock.cc:648] HybridClock initialized: now 1781996578195208 us; error 59 us; skew 500 ppm
I20260620 23:02:58.195637 20019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:58.197911 20019 webserver.cc:460] Webserver started at http://127.10.134.190:36009/ using document root <none> and password file <none>
I20260620 23:02:58.198629 20019 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:58.198773 20019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:58.199067 20019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:58.201737 20019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "41fea6cf51404172a2c1216bfe0964b1"
format_stamp: "Formatted at 2026-06-20 23:02:58 on dist-test-slave-8k25"
I20260620 23:02:58.202508 20019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "41fea6cf51404172a2c1216bfe0964b1"
format_stamp: "Formatted at 2026-06-20 23:02:58 on dist-test-slave-8k25"
I20260620 23:02:58.207479 20019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:02:58.210809 20033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:58.212013 20019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:02:58.212188 20019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
uuid: "41fea6cf51404172a2c1216bfe0964b1"
format_stamp: "Formatted at 2026-06-20 23:02:58 on dist-test-slave-8k25"
I20260620 23:02:58.212371 20019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:58.224627 20019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:58.225371 20019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:58.225673 20019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:58.243047 20019 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.190:44973
I20260620 23:02:58.243119 20084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.190:44973 every 8 connection(s)
I20260620 23:02:58.244349 20019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260620 23:02:58.247990 20085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:58.254113 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 20019
I20260620 23:02:58.254272 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260620 23:02:58.258541 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1: Bootstrap starting.
I20260620 23:02:58.261693 20085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:58.262748 20085 log.cc:824] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:58.265527 20085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1: No bootstrap required, opened a new log
I20260620 23:02:58.273782 20085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fea6cf51404172a2c1216bfe0964b1" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44973 } }
I20260620 23:02:58.274209 20085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:58.274315 20085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fea6cf51404172a2c1216bfe0964b1, State: Initialized, Role: FOLLOWER
I20260620 23:02:58.274863 20085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fea6cf51404172a2c1216bfe0964b1" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44973 } }
I20260620 23:02:58.275138 20085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:58.275275 20085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:58.275447 20085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:58.277469 20085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fea6cf51404172a2c1216bfe0964b1" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44973 } }
I20260620 23:02:58.278002 20085 leader_election.cc:302] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41fea6cf51404172a2c1216bfe0964b1; no voters:
I20260620 23:02:58.278666 20085 leader_election.cc:288] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:58.278733 20090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:58.279580 20090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [term 1 LEADER]: Becoming Leader. State: Replica: 41fea6cf51404172a2c1216bfe0964b1, State: Running, Role: LEADER
I20260620 23:02:58.280100 20090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fea6cf51404172a2c1216bfe0964b1" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44973 } }
I20260620 23:02:58.280771 20085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:58.283018 20092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41fea6cf51404172a2c1216bfe0964b1. Latest consensus state: current_term: 1 leader_uuid: "41fea6cf51404172a2c1216bfe0964b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fea6cf51404172a2c1216bfe0964b1" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44973 } } }
I20260620 23:02:58.284101 20092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:58.283821 20091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41fea6cf51404172a2c1216bfe0964b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41fea6cf51404172a2c1216bfe0964b1" member_type: VOTER last_known_addr { host: "127.10.134.190" port: 44973 } } }
I20260620 23:02:58.284986 20091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:58.286142 20096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:58.291675 20096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:58.299535 20096 catalog_manager.cc:1347] Generated new cluster ID: c23011d0753d4b68996f283e0a4789a6
I20260620 23:02:58.299642 20096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:58.323473 20096 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:58.324952 20096 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:58.332660 20096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41fea6cf51404172a2c1216bfe0964b1: Generated new TSK 0
I20260620 23:02:58.333479 20096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:58.341349 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.129:0
--local_ip_for_outbound_sockets=127.10.134.129
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44973
--builtin_ntp_servers=127.10.134.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260620 23:02:58.674407 20109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:58.674679 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:58.674793 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:58.674911 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:58.682112 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:58.682288 20109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.129
I20260620 23:02:58.691215 20109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.10.134.129
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.129
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:58.692395 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:58.694147 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:58.706513 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:58.706601 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:58.706547 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:58.707806 20109 server_base.cc:1034] running on GCE node
I20260620 23:02:58.708338 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:58.709142 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:58.710340 20109 hybrid_clock.cc:648] HybridClock initialized: now 1781996578710282 us; error 50 us; skew 500 ppm
I20260620 23:02:58.710669 20109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:58.712791 20109 webserver.cc:460] Webserver started at http://127.10.134.129:41801/ using document root <none> and password file <none>
I20260620 23:02:58.713459 20109 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:58.713587 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:58.713887 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:58.716513 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "74332d0b4abb44cd84822c9f09f1695e"
format_stamp: "Formatted at 2026-06-20 23:02:58 on dist-test-slave-8k25"
I20260620 23:02:58.717289 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "74332d0b4abb44cd84822c9f09f1695e"
format_stamp: "Formatted at 2026-06-20 23:02:58 on dist-test-slave-8k25"
I20260620 23:02:58.722210 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:02:58.725482 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:58.726868 20109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:58.727049 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "74332d0b4abb44cd84822c9f09f1695e"
format_stamp: "Formatted at 2026-06-20 23:02:58 on dist-test-slave-8k25"
I20260620 23:02:58.727274 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:58.743355 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:58.744107 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:58.744403 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:58.745265 20109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:58.746659 20109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:58.746779 20109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:58.746912 20109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:58.746999 20109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:58.780889 20109 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.129:38387
I20260620 23:02:58.780925 20235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.129:38387 every 8 connection(s)
I20260620 23:02:58.782361 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260620 23:02:58.787659 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 20109
I20260620 23:02:58.787829 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260620 23:02:58.791234 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.130:0
--local_ip_for_outbound_sockets=127.10.134.130
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44973
--builtin_ntp_servers=127.10.134.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:58.796967 20236 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44973
I20260620 23:02:58.797292 20236 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:58.798193 20236 heartbeater.cc:511] Master 127.10.134.190:44973 requested a full tablet report, sending...
I20260620 23:02:58.800354 20050 ts_manager.cc:194] Registered new tserver with Master: 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387)
I20260620 23:02:58.801863 20050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.129:37223
W20260620 23:02:59.108119 20240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:59.108412 20240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:59.108469 20240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:59.108536 20240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:59.115542 20240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:59.115674 20240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.130
I20260620 23:02:59.124349 20240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.10.134.130
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.130
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:59.125510 20240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:59.127118 20240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:59.139976 20245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:59.139992 20248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:59.140060 20246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:59.140691 20240 server_base.cc:1034] running on GCE node
I20260620 23:02:59.141193 20240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:59.142063 20240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:59.143267 20240 hybrid_clock.cc:648] HybridClock initialized: now 1781996579143220 us; error 42 us; skew 500 ppm
I20260620 23:02:59.143611 20240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:59.145808 20240 webserver.cc:460] Webserver started at http://127.10.134.130:40589/ using document root <none> and password file <none>
I20260620 23:02:59.146562 20240 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:59.146684 20240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:59.147007 20240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:59.149530 20240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b2735e06e18f4d01b65a44cd8436fd51"
format_stamp: "Formatted at 2026-06-20 23:02:59 on dist-test-slave-8k25"
I20260620 23:02:59.150308 20240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b2735e06e18f4d01b65a44cd8436fd51"
format_stamp: "Formatted at 2026-06-20 23:02:59 on dist-test-slave-8k25"
I20260620 23:02:59.155162 20240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:02:59.158499 20254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:59.159754 20240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 23:02:59.159925 20240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b2735e06e18f4d01b65a44cd8436fd51"
format_stamp: "Formatted at 2026-06-20 23:02:59 on dist-test-slave-8k25"
I20260620 23:02:59.160128 20240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:59.174423 20240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:59.175205 20240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:59.175518 20240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:59.176406 20240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:59.177827 20240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:59.177944 20240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:59.178107 20240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:59.178196 20240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:59.210665 20240 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.130:38265
I20260620 23:02:59.210716 20366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.130:38265 every 8 connection(s)
I20260620 23:02:59.211974 20240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260620 23:02:59.218010 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 20240
I20260620 23:02:59.218171 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260620 23:02:59.221562 10778 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
/tmp/dist-test-taskwogBeZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.134.131:0
--local_ip_for_outbound_sockets=127.10.134.131
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44973
--builtin_ntp_servers=127.10.134.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260620 23:02:59.227674 20367 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44973
I20260620 23:02:59.228008 20367 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:59.228880 20367 heartbeater.cc:511] Master 127.10.134.190:44973 requested a full tablet report, sending...
I20260620 23:02:59.230471 20050 ts_manager.cc:194] Registered new tserver with Master: b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130:38265)
I20260620 23:02:59.231716 20050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.130:42289
W20260620 23:02:59.545092 20370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:02:59.545367 20370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:59.545491 20370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:59.545617 20370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:59.552757 20370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:59.552945 20370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.134.131
I20260620 23:02:59.561753 20370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.134.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.134.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.10.134.131
--webserver_port=0
--tserver_master_addrs=127.10.134.190:44973
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.134.131
--log_dir=/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:06:24 UTC on ccycloud.quasar-oxjrdg.root.comops.site
build id 4457794
ASAN enabled
I20260620 23:02:59.563040 20370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:59.564906 20370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:59.577507 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:59.577538 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:59.577574 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:59.578246 20370 server_base.cc:1034] running on GCE node
I20260620 23:02:59.578822 20370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:59.579638 20370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:59.580832 20370 hybrid_clock.cc:648] HybridClock initialized: now 1781996579580787 us; error 45 us; skew 500 ppm
I20260620 23:02:59.581147 20370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:02:59.583451 20370 webserver.cc:460] Webserver started at http://127.10.134.131:39859/ using document root <none> and password file <none>
I20260620 23:02:59.584079 20370 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:59.584160 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:59.584417 20370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:59.586999 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
format_stamp: "Formatted at 2026-06-20 23:02:59 on dist-test-slave-8k25"
I20260620 23:02:59.587802 20370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
format_stamp: "Formatted at 2026-06-20 23:02:59 on dist-test-slave-8k25"
I20260620 23:02:59.592787 20370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:02:59.596030 20384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:59.597303 20370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:02:59.597483 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
format_stamp: "Formatted at 2026-06-20 23:02:59 on dist-test-slave-8k25"
I20260620 23:02:59.597689 20370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:59.612612 20370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:59.613461 20370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:59.613754 20370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:59.614734 20370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:59.616155 20370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:59.616277 20370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:59.616418 20370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:59.616506 20370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:59.648512 20370 rpc_server.cc:307] RPC server started. Bound to: 127.10.134.131:33165
I20260620 23:02:59.648566 20496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.134.131:33165 every 8 connection(s)
I20260620 23:02:59.649796 20370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260620 23:02:59.660029 10778 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu as pid 20370
I20260620 23:02:59.660229 10778 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260620 23:02:59.663323 20497 heartbeater.cc:347] Connected to a master server at 127.10.134.190:44973
I20260620 23:02:59.663676 20497 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:59.664350 20497 heartbeater.cc:511] Master 127.10.134.190:44973 requested a full tablet report, sending...
I20260620 23:02:59.665714 20050 ts_manager.cc:194] Registered new tserver with Master: 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:02:59.666857 20050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.134.131:56167
I20260620 23:02:59.675604 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:59.690904 10778 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:59.690989 10778 test_util.cc:268] Using random seed: 1763678688
I20260620 23:02:59.709221 20050 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60196:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:02:59.711206 20050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:59.746958 20162 tablet_service.cc:1467] Processing CreateTablet for tablet e34c642f9917492c8b092a4960da1ce2 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 23:02:59.748426 20167 tablet_service.cc:1467] Processing CreateTablet for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 23:02:59.748914 20162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34c642f9917492c8b092a4960da1ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.749192 20166 tablet_service.cc:1467] Processing CreateTablet for tablet 0cd9e48c0938411595fc8de40ffbfe7d (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 23:02:59.749341 20167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.749606 20166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cd9e48c0938411595fc8de40ffbfe7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.750409 20165 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4b75351c2b4ea99ab5aa51590c883d (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 23:02:59.751211 20164 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee56b451bda4bb0999178fb037912f2 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 23:02:59.751961 20163 tablet_service.cc:1467] Processing CreateTablet for tablet 11647cd1de234166a599abc7550b4940 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 23:02:59.747375 20169 tablet_service.cc:1467] Processing CreateTablet for tablet 11683ac997aa42a7b39fe2ba49a63662 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 23:02:59.747058 20168 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1ba3b1e06b48d397e8bfaef8712a76 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 23:02:59.747030 20171 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9dce672bbe4f618869b41d13db2539 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 23:02:59.754130 20165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4b75351c2b4ea99ab5aa51590c883d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.758296 20168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1ba3b1e06b48d397e8bfaef8712a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.761812 20163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11647cd1de234166a599abc7550b4940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.763159 20171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9dce672bbe4f618869b41d13db2539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.747915 20170 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9e300162904071a0010df25b78f24e (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 23:02:59.764263 20170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9e300162904071a0010df25b78f24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.766085 20164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee56b451bda4bb0999178fb037912f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.760305 20297 tablet_service.cc:1467] Processing CreateTablet for tablet 0cd9e48c0938411595fc8de40ffbfe7d (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 23:02:59.757891 20430 tablet_service.cc:1467] Processing CreateTablet for tablet 11683ac997aa42a7b39fe2ba49a63662 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 23:02:59.767509 20297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cd9e48c0938411595fc8de40ffbfe7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.768136 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11683ac997aa42a7b39fe2ba49a63662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.745112 20429 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1ba3b1e06b48d397e8bfaef8712a76 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 23:02:59.770087 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11683ac997aa42a7b39fe2ba49a63662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.759840 20298 tablet_service.cc:1467] Processing CreateTablet for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 23:02:59.770284 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1ba3b1e06b48d397e8bfaef8712a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.770478 20298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.757692 20431 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9e300162904071a0010df25b78f24e (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 23:02:59.771052 20431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9e300162904071a0010df25b78f24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.761205 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee56b451bda4bb0999178fb037912f2 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 23:02:59.771507 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee56b451bda4bb0999178fb037912f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.761647 20294 tablet_service.cc:1467] Processing CreateTablet for tablet 11647cd1de234166a599abc7550b4940 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 23:02:59.753835 20427 tablet_service.cc:1467] Processing CreateTablet for tablet 0cd9e48c0938411595fc8de40ffbfe7d (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260620 23:02:59.772485 20294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11647cd1de234166a599abc7550b4940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.772536 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cd9e48c0938411595fc8de40ffbfe7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.762933 20423 tablet_service.cc:1467] Processing CreateTablet for tablet e34c642f9917492c8b092a4960da1ce2 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 23:02:59.759372 20299 tablet_service.cc:1467] Processing CreateTablet for tablet 8c1ba3b1e06b48d397e8bfaef8712a76 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260620 23:02:59.773252 20423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34c642f9917492c8b092a4960da1ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.773586 20299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c1ba3b1e06b48d397e8bfaef8712a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.762312 20424 tablet_service.cc:1467] Processing CreateTablet for tablet 11647cd1de234166a599abc7550b4940 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260620 23:02:59.773993 20424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11647cd1de234166a599abc7550b4940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.760776 20296 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4b75351c2b4ea99ab5aa51590c883d (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 23:02:59.762111 20428 tablet_service.cc:1467] Processing CreateTablet for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260620 23:02:59.774501 20296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4b75351c2b4ea99ab5aa51590c883d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.774736 20428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.762132 20293 tablet_service.cc:1467] Processing CreateTablet for tablet e34c642f9917492c8b092a4960da1ce2 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260620 23:02:59.745483 20426 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4b75351c2b4ea99ab5aa51590c883d (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260620 23:02:59.775430 20426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4b75351c2b4ea99ab5aa51590c883d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.775431 20293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34c642f9917492c8b092a4960da1ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.762657 20432 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9dce672bbe4f618869b41d13db2539 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 23:02:59.758926 20300 tablet_service.cc:1467] Processing CreateTablet for tablet 11683ac997aa42a7b39fe2ba49a63662 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260620 23:02:59.776122 20432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9dce672bbe4f618869b41d13db2539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.776331 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11683ac997aa42a7b39fe2ba49a63662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.758404 20301 tablet_service.cc:1467] Processing CreateTablet for tablet 4f9e300162904071a0010df25b78f24e (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260620 23:02:59.777644 20301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f9e300162904071a0010df25b78f24e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.757198 20302 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9dce672bbe4f618869b41d13db2539 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260620 23:02:59.786198 20302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9dce672bbe4f618869b41d13db2539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.745164 20425 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee56b451bda4bb0999178fb037912f2 (DEFAULT_TABLE table=test-workload [id=c4d3e1ede2004b389d4bdc3d802391c8]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260620 23:02:59.805526 20236 heartbeater.cc:503] Master 127.10.134.190:44973 was elected leader, sending a full tablet report...
I20260620 23:02:59.805754 20425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee56b451bda4bb0999178fb037912f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:59.862955 20522 tablet_bootstrap.cc:492] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.868521 20522 tablet_bootstrap.cc:654] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.870096 20522 log.cc:824] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:59.871412 20521 tablet_bootstrap.cc:492] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.874405 20521 tablet_bootstrap.cc:654] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.875576 20521 log.cc:824] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:59.876801 20522 tablet_bootstrap.cc:492] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.877223 20522 ts_tablet_manager.cc:1397] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260620 23:02:59.884487 20521 tablet_bootstrap.cc:492] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.884769 20521 ts_tablet_manager.cc:1397] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260620 23:02:59.886088 20520 tablet_bootstrap.cc:492] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.889218 20520 tablet_bootstrap.cc:654] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.889302 20522 raft_consensus.cc:348] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.889737 20522 raft_consensus.cc:729] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.890353 20520 log.cc:824] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:59.890235 20522 consensus_queue.cc:260] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.891499 20497 heartbeater.cc:503] Master 127.10.134.190:44973 was elected leader, sending a full tablet report...
I20260620 23:02:59.891987 20522 ts_tablet_manager.cc:1428] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260620 23:02:59.892385 20522 tablet_bootstrap.cc:492] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.894034 20522 tablet_bootstrap.cc:654] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.895172 20521 raft_consensus.cc:348] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.895561 20521 raft_consensus.cc:729] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.896039 20522 tablet_bootstrap.cc:492] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.896196 20522 ts_tablet_manager.cc:1397] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:02:59.896005 20521 consensus_queue.cc:260] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.897238 20522 raft_consensus.cc:348] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.897548 20522 raft_consensus.cc:729] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.897737 20522 consensus_queue.cc:260] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.898384 20522 ts_tablet_manager.cc:1428] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:02:59.898665 20522 tablet_bootstrap.cc:492] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
W20260620 23:02:59.901466 20498 tablet.cc:2369] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:59.901746 20522 tablet_bootstrap.cc:654] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.906177 20520 tablet_bootstrap.cc:492] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:02:59.906476 20520 ts_tablet_manager.cc:1397] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20260620 23:02:59.910722 20367 heartbeater.cc:503] Master 127.10.134.190:44973 was elected leader, sending a full tablet report...
I20260620 23:02:59.911496 20521 ts_tablet_manager.cc:1428] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.027s user 0.012s sys 0.013s
I20260620 23:02:59.910488 20522 tablet_bootstrap.cc:492] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.911762 20522 ts_tablet_manager.cc:1397] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260620 23:02:59.912117 20521 tablet_bootstrap.cc:492] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.912457 20522 raft_consensus.cc:348] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.912725 20522 raft_consensus.cc:729] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.912923 20522 consensus_queue.cc:260] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.913482 20522 ts_tablet_manager.cc:1428] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:02:59.913841 20522 tablet_bootstrap.cc:492] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.915113 20521 tablet_bootstrap.cc:654] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.915447 20522 tablet_bootstrap.cc:654] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.915604 20520 raft_consensus.cc:348] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.916093 20520 raft_consensus.cc:729] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.916638 20520 consensus_queue.cc:260] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.927539 20522 tablet_bootstrap.cc:492] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.927754 20522 ts_tablet_manager.cc:1397] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260620 23:02:59.928213 20521 tablet_bootstrap.cc:492] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.928377 20521 ts_tablet_manager.cc:1397] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.012s
I20260620 23:02:59.928663 20520 ts_tablet_manager.cc:1428] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s
I20260620 23:02:59.928802 20522 raft_consensus.cc:348] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.929056 20520 tablet_bootstrap.cc:492] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.929160 20522 raft_consensus.cc:729] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.929265 20521 raft_consensus.cc:348] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.929529 20521 raft_consensus.cc:729] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.929384 20522 consensus_queue.cc:260] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.929751 20521 consensus_queue.cc:260] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.930254 20522 ts_tablet_manager.cc:1428] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:02:59.930545 20522 tablet_bootstrap.cc:492] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.930804 20520 tablet_bootstrap.cc:654] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.930827 20521 ts_tablet_manager.cc:1428] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:02:59.931174 20521 tablet_bootstrap.cc:492] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.932587 20522 tablet_bootstrap.cc:654] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.932859 20521 tablet_bootstrap.cc:654] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.943718 20520 tablet_bootstrap.cc:492] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:02:59.943893 20520 ts_tablet_manager.cc:1397] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260620 23:02:59.944279 20522 tablet_bootstrap.cc:492] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.944293 20521 tablet_bootstrap.cc:492] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.944442 20521 ts_tablet_manager.cc:1397] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260620 23:02:59.944451 20522 ts_tablet_manager.cc:1397] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20260620 23:02:59.944820 20520 raft_consensus.cc:348] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.945070 20520 raft_consensus.cc:729] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.944993 20522 raft_consensus.cc:348] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:02:59.945117 20521 raft_consensus.cc:348] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.945307 20522 raft_consensus.cc:729] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.945344 20521 raft_consensus.cc:729] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.945290 20520 consensus_queue.cc:260] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.945489 20522 consensus_queue.cc:260] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:02:59.945504 20521 consensus_queue.cc:260] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.945904 20520 ts_tablet_manager.cc:1428] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:02:59.946058 20521 ts_tablet_manager.cc:1428] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:59.946194 20520 tablet_bootstrap.cc:492] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.946401 20521 tablet_bootstrap.cc:492] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.947819 20520 tablet_bootstrap.cc:654] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.947865 20521 tablet_bootstrap.cc:654] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.947983 20522 ts_tablet_manager.cc:1428] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:02:59.950073 20522 tablet_bootstrap.cc:492] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.951730 20522 tablet_bootstrap.cc:654] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.956135 20521 tablet_bootstrap.cc:492] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.956295 20521 ts_tablet_manager.cc:1397] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 23:02:59.956398 20520 tablet_bootstrap.cc:492] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:02:59.956557 20520 ts_tablet_manager.cc:1397] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260620 23:02:59.957031 20521 raft_consensus.cc:348] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.957288 20521 raft_consensus.cc:729] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.957187 20520 raft_consensus.cc:348] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.957403 20520 raft_consensus.cc:729] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.957460 20521 consensus_queue.cc:260] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.957566 20520 consensus_queue.cc:260] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.958199 20520 ts_tablet_manager.cc:1428] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:02:59.958506 20520 tablet_bootstrap.cc:492] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.958850 20521 ts_tablet_manager.cc:1428] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:02:59.959107 20521 tablet_bootstrap.cc:492] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.960144 20520 tablet_bootstrap.cc:654] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.960719 20521 tablet_bootstrap.cc:654] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.962250 20522 tablet_bootstrap.cc:492] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.962422 20522 ts_tablet_manager.cc:1397] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s
I20260620 23:02:59.962646 20521 tablet_bootstrap.cc:492] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.962802 20521 ts_tablet_manager.cc:1397] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 23:02:59.963284 20522 raft_consensus.cc:348] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.963554 20522 raft_consensus.cc:729] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.963464 20521 raft_consensus.cc:348] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.963725 20521 raft_consensus.cc:729] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.963797 20522 consensus_queue.cc:260] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
W20260620 23:02:59.964171 20368 tablet.cc:2369] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:59.964061 20521 consensus_queue.cc:260] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.964430 20522 ts_tablet_manager.cc:1428] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:59.964533 20521 ts_tablet_manager.cc:1428] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:02:59.964682 20522 tablet_bootstrap.cc:492] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.964843 20521 tablet_bootstrap.cc:492] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.966442 20521 tablet_bootstrap.cc:654] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.966673 20522 tablet_bootstrap.cc:654] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.968310 20520 tablet_bootstrap.cc:492] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:02:59.968487 20520 ts_tablet_manager.cc:1397] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260620 23:02:59.969197 20520 raft_consensus.cc:348] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.969422 20520 raft_consensus.cc:729] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.969609 20520 consensus_queue.cc:260] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.970168 20520 ts_tablet_manager.cc:1428] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:59.970428 20520 tablet_bootstrap.cc:492] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.972012 20520 tablet_bootstrap.cc:654] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.975327 20521 tablet_bootstrap.cc:492] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.975512 20521 ts_tablet_manager.cc:1397] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260620 23:02:59.975561 20522 tablet_bootstrap.cc:492] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.975860 20522 ts_tablet_manager.cc:1397] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260620 23:02:59.976323 20521 raft_consensus.cc:348] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.976536 20521 raft_consensus.cc:729] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.976751 20521 consensus_queue.cc:260] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.977008 20522 raft_consensus.cc:348] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.977363 20521 ts_tablet_manager.cc:1428] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:02:59.977341 20522 raft_consensus.cc:729] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.977640 20521 tablet_bootstrap.cc:492] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:02:59.977576 20522 consensus_queue.cc:260] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.978367 20522 ts_tablet_manager.cc:1428] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:59.978731 20522 tablet_bootstrap.cc:492] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.981062 20522 tablet_bootstrap.cc:654] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.982170 20521 tablet_bootstrap.cc:654] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.982395 20520 tablet_bootstrap.cc:492] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:02:59.982571 20520 ts_tablet_manager.cc:1397] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260620 23:02:59.983309 20520 raft_consensus.cc:348] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.983532 20520 raft_consensus.cc:729] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.983705 20520 consensus_queue.cc:260] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.984184 20522 tablet_bootstrap.cc:492] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:02:59.984234 20520 ts_tablet_manager.cc:1428] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:02:59.984367 20522 ts_tablet_manager.cc:1397] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 23:02:59.984509 20520 tablet_bootstrap.cc:492] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.985235 20522 raft_consensus.cc:348] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.985524 20522 raft_consensus.cc:729] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.985720 20522 consensus_queue.cc:260] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.986155 20520 tablet_bootstrap.cc:654] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.986343 20522 ts_tablet_manager.cc:1428] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:59.986593 20522 tablet_bootstrap.cc:492] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:02:59.989167 20522 tablet_bootstrap.cc:654] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.991237 20521 tablet_bootstrap.cc:492] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:02:59.991425 20521 ts_tablet_manager.cc:1397] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20260620 23:02:59.992465 20521 raft_consensus.cc:348] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.992805 20521 raft_consensus.cc:729] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.993017 20521 consensus_queue.cc:260] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.994100 20520 tablet_bootstrap.cc:492] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:02:59.994256 20520 ts_tablet_manager.cc:1397] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260620 23:02:59.994982 20520 raft_consensus.cc:348] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.995227 20520 raft_consensus.cc:729] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:02:59.995419 20520 consensus_queue.cc:260] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:02:59.996057 20520 ts_tablet_manager.cc:1428] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:59.996317 20520 tablet_bootstrap.cc:492] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:02:59.996464 20521 ts_tablet_manager.cc:1428] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 23:02:59.998061 20520 tablet_bootstrap.cc:654] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:59.999275 20521 tablet_bootstrap.cc:492] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:03:00.000173 20522 tablet_bootstrap.cc:492] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:03:00.000345 20522 ts_tablet_manager.cc:1397] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260620 23:03:00.000639 20520 tablet_bootstrap.cc:492] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:03:00.000805 20520 ts_tablet_manager.cc:1397] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:03:00.001004 20521 tablet_bootstrap.cc:654] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:00.001157 20522 raft_consensus.cc:348] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.001432 20522 raft_consensus.cc:729] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.001483 20520 raft_consensus.cc:348] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.001714 20520 raft_consensus.cc:729] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.001636 20522 consensus_queue.cc:260] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.001893 20520 consensus_queue.cc:260] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.002296 20522 ts_tablet_manager.cc:1428] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 23:03:00.002602 20522 tablet_bootstrap.cc:492] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8: Bootstrap starting.
I20260620 23:03:00.004252 20522 tablet_bootstrap.cc:654] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:00.005100 20520 ts_tablet_manager.cc:1428] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:03:00.007754 20520 tablet_bootstrap.cc:492] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:03:00.008121 20521 tablet_bootstrap.cc:492] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:03:00.008319 20521 ts_tablet_manager.cc:1397] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260620 23:03:00.008729 20522 tablet_bootstrap.cc:492] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8: No bootstrap required, opened a new log
I20260620 23:03:00.008876 20522 ts_tablet_manager.cc:1397] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 23:03:00.009122 20521 raft_consensus.cc:348] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.009358 20520 tablet_bootstrap.cc:654] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:00.009371 20521 raft_consensus.cc:729] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.009555 20521 consensus_queue.cc:260] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.009579 20522 raft_consensus.cc:348] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.009819 20522 raft_consensus.cc:729] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c653f3f085499cb5428c4ceb1e6cf8, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.010030 20522 consensus_queue.cc:260] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.010574 20522 ts_tablet_manager.cc:1428] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:03:00.011406 20521 ts_tablet_manager.cc:1428] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:03:00.011683 20521 tablet_bootstrap.cc:492] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:03:00.013260 20521 tablet_bootstrap.cc:654] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:00.015159 20521 tablet_bootstrap.cc:492] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:03:00.015471 20521 ts_tablet_manager.cc:1397] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 23:03:00.016006 20521 raft_consensus.cc:348] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:03:00.016182 20521 raft_consensus.cc:729] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.016297 20521 consensus_queue.cc:260] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:03:00.016789 20521 ts_tablet_manager.cc:1428] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:03:00.017077 20521 tablet_bootstrap.cc:492] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Bootstrap starting.
I20260620 23:03:00.018662 20521 tablet_bootstrap.cc:654] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:00.020277 20520 tablet_bootstrap.cc:492] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:03:00.020431 20520 ts_tablet_manager.cc:1397] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260620 23:03:00.021131 20520 raft_consensus.cc:348] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.021350 20520 raft_consensus.cc:729] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.021531 20520 consensus_queue.cc:260] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.022099 20520 ts_tablet_manager.cc:1428] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:03:00.022328 20520 tablet_bootstrap.cc:492] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:03:00.024323 20520 tablet_bootstrap.cc:654] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:00.027945 20521 tablet_bootstrap.cc:492] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: No bootstrap required, opened a new log
I20260620 23:03:00.028070 20521 ts_tablet_manager.cc:1397] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260620 23:03:00.028861 20521 raft_consensus.cc:348] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.029102 20521 raft_consensus.cc:729] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.029335 20521 consensus_queue.cc:260] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.029754 20520 tablet_bootstrap.cc:492] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:03:00.029902 20520 ts_tablet_manager.cc:1397] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260620 23:03:00.030249 20521 ts_tablet_manager.cc:1428] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:03:00.030643 20520 raft_consensus.cc:348] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:03:00.030875 20520 raft_consensus.cc:729] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.031050 20520 consensus_queue.cc:260] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:03:00.031728 20520 ts_tablet_manager.cc:1428] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:03:00.032028 20520 tablet_bootstrap.cc:492] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e: Bootstrap starting.
I20260620 23:03:00.033351 20520 tablet_bootstrap.cc:654] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e: Neither blocks nor log segments found. Creating new log.
W20260620 23:03:00.034641 20237 tablet.cc:2369] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:03:00.036427 20520 tablet_bootstrap.cc:492] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e: No bootstrap required, opened a new log
I20260620 23:03:00.036580 20520 ts_tablet_manager.cc:1397] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 23:03:00.037267 20520 raft_consensus.cc:348] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.037477 20520 raft_consensus.cc:729] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74332d0b4abb44cd84822c9f09f1695e, State: Initialized, Role: FOLLOWER
I20260620 23:03:00.037662 20520 consensus_queue.cc:260] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.038187 20520 ts_tablet_manager.cc:1428] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 23:03:00.074601 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f9e300162904071a0010df25b78f24e"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.074900 20322 raft_consensus.cc:482] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.075042 20322 raft_consensus.cc:3037] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.077005 20322 raft_consensus.cc:504] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.078490 20322 leader_election.cc:288] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.086844 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9e300162904071a0010df25b78f24e" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.086899 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f9e300162904071a0010df25b78f24e" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.087299 20452 raft_consensus.cc:3037] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.087299 20191 raft_consensus.cc:3037] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.090489 20191 raft_consensus.cc:2454] T 4f9e300162904071a0010df25b78f24e P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.090488 20452 raft_consensus.cc:2454] T 4f9e300162904071a0010df25b78f24e P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.091200 20257 leader_election.cc:302] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 51c653f3f085499cb5428c4ceb1e6cf8, 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.091665 20528 raft_consensus.cc:2781] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.092270 20528 raft_consensus.cc:686] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.092829 20528 consensus_queue.cc:237] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.097276 20049 catalog_manager.cc:5495] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.109885 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a9dce672bbe4f618869b41d13db2539"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.110139 20322 raft_consensus.cc:482] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.110234 20322 raft_consensus.cc:3037] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.111986 20322 raft_consensus.cc:504] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:03:00.112488 20322 leader_election.cc:288] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165), 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387)
I20260620 23:03:00.112986 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9dce672bbe4f618869b41d13db2539" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.113010 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9dce672bbe4f618869b41d13db2539" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.113240 20191 raft_consensus.cc:3037] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.113240 20452 raft_consensus.cc:3037] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.116034 20452 raft_consensus.cc:2454] T 7a9dce672bbe4f618869b41d13db2539 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.116027 20191 raft_consensus.cc:2454] T 7a9dce672bbe4f618869b41d13db2539 P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.116429 20257 leader_election.cc:302] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.116707 20528 raft_consensus.cc:2781] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.116890 20528 raft_consensus.cc:686] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.117134 20528 consensus_queue.cc:237] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } }
I20260620 23:03:00.119768 20049 catalog_manager.cc:5495] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } }
I20260620 23:03:00.128851 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c1ba3b1e06b48d397e8bfaef8712a76"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.129055 20322 raft_consensus.cc:482] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.129199 20322 raft_consensus.cc:3037] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.131207 20322 raft_consensus.cc:504] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.131840 20322 leader_election.cc:288] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.132333 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1ba3b1e06b48d397e8bfaef8712a76" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.132375 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c1ba3b1e06b48d397e8bfaef8712a76" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.132557 20191 raft_consensus.cc:3037] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.132624 20452 raft_consensus.cc:3037] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.134652 20191 raft_consensus.cc:2454] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.134802 20452 raft_consensus.cc:2454] T 8c1ba3b1e06b48d397e8bfaef8712a76 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.135047 20257 leader_election.cc:302] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.135402 20528 raft_consensus.cc:2781] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.135632 20528 raft_consensus.cc:686] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.135915 20528 consensus_queue.cc:237] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.138703 20049 catalog_manager.cc:5495] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.148005 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0cd9e48c0938411595fc8de40ffbfe7d"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.148208 20322 raft_consensus.cc:482] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.148304 20322 raft_consensus.cc:3037] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.150166 20322 raft_consensus.cc:504] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.150704 20322 leader_election.cc:288] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.151225 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cd9e48c0938411595fc8de40ffbfe7d" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.151249 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cd9e48c0938411595fc8de40ffbfe7d" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.151456 20452 raft_consensus.cc:3037] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.151499 20191 raft_consensus.cc:3037] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.153465 20452 raft_consensus.cc:2454] T 0cd9e48c0938411595fc8de40ffbfe7d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.153896 20256 leader_election.cc:302] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c653f3f085499cb5428c4ceb1e6cf8, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.153986 20191 raft_consensus.cc:2454] T 0cd9e48c0938411595fc8de40ffbfe7d P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.154239 20528 raft_consensus.cc:2781] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.154403 20528 raft_consensus.cc:686] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.154636 20528 consensus_queue.cc:237] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.157061 20049 catalog_manager.cc:5495] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.166908 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ee56b451bda4bb0999178fb037912f2"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.167114 20322 raft_consensus.cc:482] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.167208 20322 raft_consensus.cc:3037] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.168948 20322 raft_consensus.cc:504] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.169409 20322 leader_election.cc:288] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.169905 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee56b451bda4bb0999178fb037912f2" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.169907 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee56b451bda4bb0999178fb037912f2" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.170148 20452 raft_consensus.cc:3037] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.170173 20191 raft_consensus.cc:3037] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.172099 20191 raft_consensus.cc:2454] T 6ee56b451bda4bb0999178fb037912f2 P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.172152 20452 raft_consensus.cc:2454] T 6ee56b451bda4bb0999178fb037912f2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.172549 20257 leader_election.cc:302] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.172856 20528 raft_consensus.cc:2781] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.173025 20528 raft_consensus.cc:686] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.173272 20528 consensus_queue.cc:237] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.175827 20049 catalog_manager.cc:5495] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.186326 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e34c642f9917492c8b092a4960da1ce2"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.186561 20322 raft_consensus.cc:482] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.186666 20322 raft_consensus.cc:3037] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.188589 20322 raft_consensus.cc:504] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.189169 20322 leader_election.cc:288] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.189587 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34c642f9917492c8b092a4960da1ce2" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.189594 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34c642f9917492c8b092a4960da1ce2" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.189827 20191 raft_consensus.cc:3037] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.189827 20452 raft_consensus.cc:3037] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.192641 20191 raft_consensus.cc:2454] T e34c642f9917492c8b092a4960da1ce2 P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.192632 20452 raft_consensus.cc:2454] T e34c642f9917492c8b092a4960da1ce2 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.193019 20256 leader_election.cc:302] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c653f3f085499cb5428c4ceb1e6cf8, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.193289 20528 raft_consensus.cc:2781] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.193471 20528 raft_consensus.cc:686] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.193708 20528 consensus_queue.cc:237] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.196416 20049 catalog_manager.cc:5495] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.206067 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a4b75351c2b4ea99ab5aa51590c883d"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.206285 20322 raft_consensus.cc:482] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.206418 20322 raft_consensus.cc:3037] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.208244 20322 raft_consensus.cc:504] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.208819 20322 leader_election.cc:288] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.209348 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4b75351c2b4ea99ab5aa51590c883d" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.209385 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4b75351c2b4ea99ab5aa51590c883d" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.209578 20452 raft_consensus.cc:3037] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.209724 20191 raft_consensus.cc:3037] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.211719 20452 raft_consensus.cc:2454] T 2a4b75351c2b4ea99ab5aa51590c883d P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.211792 20191 raft_consensus.cc:2454] T 2a4b75351c2b4ea99ab5aa51590c883d P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.212302 20257 leader_election.cc:302] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.212787 20528 raft_consensus.cc:2781] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.212937 20528 raft_consensus.cc:686] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.213187 20528 consensus_queue.cc:237] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.215854 20049 catalog_manager.cc:5495] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.232482 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11647cd1de234166a599abc7550b4940"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.232728 20322 raft_consensus.cc:482] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.232856 20322 raft_consensus.cc:3037] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.234912 20322 raft_consensus.cc:504] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.235519 20322 leader_election.cc:288] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.235935 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11647cd1de234166a599abc7550b4940" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.236012 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11647cd1de234166a599abc7550b4940" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.236215 20191 raft_consensus.cc:3037] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.236255 20452 raft_consensus.cc:3037] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.238480 20452 raft_consensus.cc:2454] T 11647cd1de234166a599abc7550b4940 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.238867 20191 raft_consensus.cc:2454] T 11647cd1de234166a599abc7550b4940 P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.238956 20256 leader_election.cc:302] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c653f3f085499cb5428c4ceb1e6cf8, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.239341 20528 raft_consensus.cc:2781] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.239539 20528 raft_consensus.cc:686] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.239802 20528 consensus_queue.cc:237] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.242533 20049 catalog_manager.cc:5495] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.253206 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11683ac997aa42a7b39fe2ba49a63662"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.253412 20322 raft_consensus.cc:482] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.253548 20322 raft_consensus.cc:3037] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.255524 20322 raft_consensus.cc:504] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.256553 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11683ac997aa42a7b39fe2ba49a63662" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.256585 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11683ac997aa42a7b39fe2ba49a63662" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.256785 20452 raft_consensus.cc:3037] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.256860 20191 raft_consensus.cc:3037] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.259606 20191 raft_consensus.cc:2454] T 11683ac997aa42a7b39fe2ba49a63662 P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.259598 20452 raft_consensus.cc:2454] T 11683ac997aa42a7b39fe2ba49a63662 P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.260005 20257 leader_election.cc:302] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.260149 20322 leader_election.cc:288] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.260339 20528 raft_consensus.cc:2781] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.260494 20528 raft_consensus.cc:686] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.260814 20528 consensus_queue.cc:237] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.264981 20049 catalog_manager.cc:5495] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.273605 20322 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db69e2f5d6f04d3d90fbc1bb2d91e5ab"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.273785 20322 raft_consensus.cc:482] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:03:00.273867 20322 raft_consensus.cc:3037] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.275709 20322 raft_consensus.cc:504] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.276238 20322 leader_election.cc:288] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Requested vote from peers 74332d0b4abb44cd84822c9f09f1695e (127.10.134.129:38387), 51c653f3f085499cb5428c4ceb1e6cf8 (127.10.134.131:33165)
I20260620 23:03:00.276734 20191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db69e2f5d6f04d3d90fbc1bb2d91e5ab" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74332d0b4abb44cd84822c9f09f1695e"
I20260620 23:03:00.276975 20191 raft_consensus.cc:3037] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.276870 20452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db69e2f5d6f04d3d90fbc1bb2d91e5ab" candidate_uuid: "b2735e06e18f4d01b65a44cd8436fd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c653f3f085499cb5428c4ceb1e6cf8"
I20260620 23:03:00.277190 20452 raft_consensus.cc:3037] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:00.279244 20191 raft_consensus.cc:2454] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 74332d0b4abb44cd84822c9f09f1695e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.279620 20452 raft_consensus.cc:2454] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P 51c653f3f085499cb5428c4ceb1e6cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2735e06e18f4d01b65a44cd8436fd51 in term 1.
I20260620 23:03:00.279707 20257 leader_election.cc:302] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74332d0b4abb44cd84822c9f09f1695e, b2735e06e18f4d01b65a44cd8436fd51; no voters:
I20260620 23:03:00.280040 20528 raft_consensus.cc:2781] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:00.280205 20528 raft_consensus.cc:686] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Leader. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.280436 20528 consensus_queue.cc:237] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.283171 20049 catalog_manager.cc:5495] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2735e06e18f4d01b65a44cd8436fd51 (127.10.134.130). New cstate: current_term: 1 leader_uuid: "b2735e06e18f4d01b65a44cd8436fd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:00.293283 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f9e300162904071a0010df25b78f24e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.294059 20571 tablet_replica.cc:330] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.294454 20571 raft_consensus.cc:2229] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:03:00.294977 20571 pending_rounds.cc:62] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.295110 20571 pending_rounds.cc:68] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992063741952 op_type: NO_OP noop_request { }
I20260620 23:03:00.295348 20571 raft_consensus.cc:2866] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.295495 20571 raft_consensus.cc:2258] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.296981 20571 ts_tablet_manager.cc:1905] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.301561 20571 ts_tablet_manager.cc:1918] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.301734 20571 log.cc:1196] T 4f9e300162904071a0010df25b78f24e P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f9e300162904071a0010df25b78f24e
I20260620 23:03:00.311714 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9dce672bbe4f618869b41d13db2539 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.311992 20571 tablet_replica.cc:330] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.312191 20571 raft_consensus.cc:2229] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:03:00.312393 20571 pending_rounds.cc:62] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.312458 20571 pending_rounds.cc:68] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992161144832 op_type: NO_OP noop_request { }
I20260620 23:03:00.312572 20571 raft_consensus.cc:2866] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.312665 20571 raft_consensus.cc:2258] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.313429 20571 ts_tablet_manager.cc:1905] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.317644 20571 ts_tablet_manager.cc:1918] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.317746 20571 log.cc:1196] T 7a9dce672bbe4f618869b41d13db2539 P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a9dce672bbe4f618869b41d13db2539
I20260620 23:03:00.327490 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c1ba3b1e06b48d397e8bfaef8712a76 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.327775 20571 tablet_replica.cc:330] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.327921 20571 raft_consensus.cc:2229] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:03:00.328133 20571 pending_rounds.cc:62] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.328186 20571 pending_rounds.cc:68] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992238616576 op_type: NO_OP noop_request { }
I20260620 23:03:00.328305 20571 raft_consensus.cc:2866] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.328382 20571 raft_consensus.cc:2258] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.329247 20571 ts_tablet_manager.cc:1905] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.333346 20571 ts_tablet_manager.cc:1918] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.333458 20571 log.cc:1196] T 8c1ba3b1e06b48d397e8bfaef8712a76 P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/8c1ba3b1e06b48d397e8bfaef8712a76
I20260620 23:03:00.343006 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd9e48c0938411595fc8de40ffbfe7d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.343262 20571 tablet_replica.cc:330] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.343425 20571 raft_consensus.cc:2229] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:03:00.343616 20571 pending_rounds.cc:62] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.343686 20571 pending_rounds.cc:68] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992314953728 op_type: NO_OP noop_request { }
I20260620 23:03:00.343791 20571 raft_consensus.cc:2866] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.343881 20571 raft_consensus.cc:2258] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.344652 20571 ts_tablet_manager.cc:1905] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.348632 20571 ts_tablet_manager.cc:1918] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.348742 20571 log.cc:1196] T 0cd9e48c0938411595fc8de40ffbfe7d P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/0cd9e48c0938411595fc8de40ffbfe7d
I20260620 23:03:00.358285 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee56b451bda4bb0999178fb037912f2 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.358557 20571 tablet_replica.cc:330] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.358706 20571 raft_consensus.cc:2229] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:03:00.358916 20571 pending_rounds.cc:62] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.358965 20571 pending_rounds.cc:68] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992391200768 op_type: NO_OP noop_request { }
I20260620 23:03:00.359076 20571 raft_consensus.cc:2866] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.359174 20571 raft_consensus.cc:2258] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.359967 20571 ts_tablet_manager.cc:1905] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.363849 20571 ts_tablet_manager.cc:1918] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.363967 20571 log.cc:1196] T 6ee56b451bda4bb0999178fb037912f2 P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/6ee56b451bda4bb0999178fb037912f2
I20260620 23:03:00.373648 20322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e34c642f9917492c8b092a4960da1ce2"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.373857 20322 raft_consensus.cc:580] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Received request to step down
I20260620 23:03:00.373982 20322 raft_consensus.cc:3032] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:03:00.374053 20322 raft_consensus.cc:729] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.374280 20322 consensus_queue.cc:260] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.374528 20322 raft_consensus.cc:3037] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:03:00.383785 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet e34c642f9917492c8b092a4960da1ce2 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.384059 20571 tablet_replica.cc:330] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.384231 20571 raft_consensus.cc:2229] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:03:00.384397 20571 pending_rounds.cc:62] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.384469 20571 pending_rounds.cc:68] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992475889664 op_type: NO_OP noop_request { }
I20260620 23:03:00.384573 20571 raft_consensus.cc:2866] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.384666 20571 raft_consensus.cc:2258] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.385449 20571 ts_tablet_manager.cc:1905] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.389829 20571 ts_tablet_manager.cc:1918] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.389977 20571 log.cc:1196] T e34c642f9917492c8b092a4960da1ce2 P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/e34c642f9917492c8b092a4960da1ce2
I20260620 23:03:00.399504 20322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a4b75351c2b4ea99ab5aa51590c883d"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.399725 20322 raft_consensus.cc:580] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Received request to step down
I20260620 23:03:00.399807 20322 raft_consensus.cc:3032] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:03:00.399856 20322 raft_consensus.cc:729] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.400022 20322 consensus_queue.cc:260] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.400233 20322 raft_consensus.cc:3037] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:03:00.409716 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4b75351c2b4ea99ab5aa51590c883d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.410061 20571 tablet_replica.cc:330] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.410234 20571 raft_consensus.cc:2229] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:03:00.410436 20571 pending_rounds.cc:62] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.410485 20571 pending_rounds.cc:68] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992555241472 op_type: NO_OP noop_request { }
I20260620 23:03:00.410673 20571 raft_consensus.cc:2866] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.410794 20571 raft_consensus.cc:2258] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.411582 20571 ts_tablet_manager.cc:1905] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.415673 20571 ts_tablet_manager.cc:1918] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.415828 20571 log.cc:1196] T 2a4b75351c2b4ea99ab5aa51590c883d P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/2a4b75351c2b4ea99ab5aa51590c883d
I20260620 23:03:00.425705 20322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11647cd1de234166a599abc7550b4940"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.425984 20322 raft_consensus.cc:580] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Received request to step down
I20260620 23:03:00.426113 20322 raft_consensus.cc:3032] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:03:00.426198 20322 raft_consensus.cc:729] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.426420 20322 consensus_queue.cc:260] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.426702 20322 raft_consensus.cc:3037] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:03:00.435971 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 11647cd1de234166a599abc7550b4940 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.436223 20571 tablet_replica.cc:330] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.436372 20571 raft_consensus.cc:2229] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:03:00.436489 20571 pending_rounds.cc:62] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.436547 20571 pending_rounds.cc:68] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992664088576 op_type: NO_OP noop_request { }
I20260620 23:03:00.436645 20571 raft_consensus.cc:2866] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.436728 20571 raft_consensus.cc:2258] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.437441 20571 ts_tablet_manager.cc:1905] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.441380 20571 ts_tablet_manager.cc:1918] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.441473 20571 log.cc:1196] T 11647cd1de234166a599abc7550b4940 P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/11647cd1de234166a599abc7550b4940
I20260620 23:03:00.450958 20322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11683ac997aa42a7b39fe2ba49a63662"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.451129 20322 raft_consensus.cc:580] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Received request to step down
I20260620 23:03:00.451205 20322 raft_consensus.cc:3032] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:03:00.451254 20322 raft_consensus.cc:729] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.451422 20322 consensus_queue.cc:260] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.451627 20322 raft_consensus.cc:3037] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:03:00.460578 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 11683ac997aa42a7b39fe2ba49a63662 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.460927 20571 tablet_replica.cc:330] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.461062 20571 raft_consensus.cc:2229] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:03:00.461266 20571 pending_rounds.cc:62] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.461335 20571 pending_rounds.cc:68] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992750219264 op_type: NO_OP noop_request { }
I20260620 23:03:00.461479 20571 raft_consensus.cc:2866] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.461557 20571 raft_consensus.cc:2258] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.462375 20571 ts_tablet_manager.cc:1905] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.466463 20571 ts_tablet_manager.cc:1918] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.466573 20571 log.cc:1196] T 11683ac997aa42a7b39fe2ba49a63662 P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/11683ac997aa42a7b39fe2ba49a63662
I20260620 23:03:00.476089 20322 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db69e2f5d6f04d3d90fbc1bb2d91e5ab"
dest_uuid: "b2735e06e18f4d01b65a44cd8436fd51"
from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.476291 20322 raft_consensus.cc:580] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Received request to step down
I20260620 23:03:00.476373 20322 raft_consensus.cc:3032] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260620 23:03:00.476421 20322 raft_consensus.cc:729] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2735e06e18f4d01b65a44cd8436fd51, State: Running, Role: LEADER
I20260620 23:03:00.476598 20322 consensus_queue.cc:260] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74332d0b4abb44cd84822c9f09f1695e" member_type: VOTER last_known_addr { host: "127.10.134.129" port: 38387 } } peers { permanent_uuid: "b2735e06e18f4d01b65a44cd8436fd51" member_type: VOTER last_known_addr { host: "127.10.134.130" port: 38265 } } peers { permanent_uuid: "51c653f3f085499cb5428c4ceb1e6cf8" member_type: VOTER last_known_addr { host: "127.10.134.131" port: 33165 } }
I20260620 23:03:00.476847 20322 raft_consensus.cc:3037] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:03:00.486150 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet db69e2f5d6f04d3d90fbc1bb2d91e5ab with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:40718
I20260620 23:03:00.486423 20571 tablet_replica.cc:330] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: stopping tablet replica
I20260620 23:03:00.486572 20571 raft_consensus.cc:2229] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260620 23:03:00.486696 20571 pending_rounds.cc:62] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Trying to abort 1 pending ops.
I20260620 23:03:00.486753 20571 pending_rounds.cc:68] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299057992830509056 op_type: NO_OP noop_request { }
I20260620 23:03:00.486858 20571 raft_consensus.cc:2866] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 23:03:00.486943 20571 raft_consensus.cc:2258] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:00.487633 20571 ts_tablet_manager.cc:1905] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260620 23:03:00.491685 20571 ts_tablet_manager.cc:1918] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260620 23:03:00.491786 20571 log.cc:1196] T db69e2f5d6f04d3d90fbc1bb2d91e5ab P b2735e06e18f4d01b65a44cd8436fd51: Deleting WAL directory at /tmp/dist-test-taskwogBeZ/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1781996381279935-10778-0/raft_consensus-itest-cluster/ts-1/wal/wals/db69e2f5d6f04d3d90fbc1bb2d91e5ab
I20260620 23:03:00.501891 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 20109
I20260620 23:03:00.508955 20231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:03:00.612919 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 20109
I20260620 23:03:00.631120 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 20240
I20260620 23:03:00.641749 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:03:00.753695 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 20240
I20260620 23:03:00.772862 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 20370
I20260620 23:03:00.779105 20492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:03:00.872944 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 20370
I20260620 23:03:00.890333 10778 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu pid 20019
I20260620 23:03:00.891430 20080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:03:00.996532 10778 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwogBeZ/build/asan/bin/kudu with pid 20019
2026-06-20T23:03:01Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (3240 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (5084 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (199694 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20260620 23:03:01.043931 10778 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~15 seconds ago