[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260921 18:43:47.779868 3218 test_util.cc:268] Using random seed: -1587141383
I20260921 18:43:47.789336 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:43:47.789414 3218 ts_itest-base.cc:113] --------------
I20260921 18:43:47.789433 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:43:47.789485 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:43:47.789532 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:43:47Z Disabled control of system clock
I20260921 18:43:47.800964 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:34233
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:34233
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260921 18:43:47.906122 3230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:43:47.906347 3230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:43:47.906378 3230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:43:47.910228 3230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:43:47.910270 3230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:43:47.910287 3230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:43:47.910302 3230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:43:47.914654 3230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:34233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:43:47.915591 3230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:43:47.916795 3230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260921 18:43:47.925005 3230 server_base.cc:1034] running on GCE node
W20260921 18:43:47.925140 3235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:47.925196 3236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:47.925031 3238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:43:47.925945 3230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:43:47.926968 3230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:43:47.928135 3230 hybrid_clock.cc:648] HybridClock initialized: now 1790016227928106 us; error 42 us; skew 500 ppm
I20260921 18:43:47.930011 3230 webserver.cc:460] Webserver started at http://127.3.36.190:39903/ using document root <none> and password file <none>
I20260921 18:43:47.930558 3230 fs_manager.cc:359] Metadata directory not provided
I20260921 18:43:47.930619 3230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:43:47.930855 3230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:43:47.932646 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "66d73670ab464974a69727e84e06425b"
format_stamp: "Formatted at 2026-09-21 18:43:47 on dist-test-slave-l4st"
I20260921 18:43:47.933190 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "66d73670ab464974a69727e84e06425b"
format_stamp: "Formatted at 2026-09-21 18:43:47 on dist-test-slave-l4st"
I20260921 18:43:47.936775 3230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260921 18:43:47.939016 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:47.940232 3230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:43:47.940364 3230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "66d73670ab464974a69727e84e06425b"
format_stamp: "Formatted at 2026-09-21 18:43:47 on dist-test-slave-l4st"
I20260921 18:43:47.940477 3230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:43:47.967993 3230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:43:47.968636 3230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:43:47.968824 3230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:43:47.975404 3230 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:34233
I20260921 18:43:47.975421 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:34233 every 8 connection(s)
I20260921 18:43:47.976325 3230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20260921 18:43:47.979295 3297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:43:47.983949 3297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b: Bootstrap starting.
I20260921 18:43:47.986284 3297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b: Neither blocks nor log segments found. Creating new log.
I20260921 18:43:47.986616 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3230
I20260921 18:43:47.986763 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20260921 18:43:47.987107 3297 log.cc:824] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b: Log is configured to *not* fsync() on all Append() calls
I20260921 18:43:47.988932 3297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b: No bootstrap required, opened a new log
I20260921 18:43:47.993002 3297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d73670ab464974a69727e84e06425b" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 34233 } }
I20260921 18:43:47.993227 3297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:43:47.993304 3297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d73670ab464974a69727e84e06425b, State: Initialized, Role: FOLLOWER
I20260921 18:43:47.993738 3297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d73670ab464974a69727e84e06425b" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 34233 } }
I20260921 18:43:47.993899 3297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:43:47.993983 3297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:43:47.994079 3297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:43:47.994825 3297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d73670ab464974a69727e84e06425b" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 34233 } }
I20260921 18:43:47.995150 3297 leader_election.cc:302] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d73670ab464974a69727e84e06425b; no voters:
I20260921 18:43:47.995469 3297 leader_election.cc:288] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:43:47.995538 3302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:43:47.995731 3302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [term 1 LEADER]: Becoming Leader. State: Replica: 66d73670ab464974a69727e84e06425b, State: Running, Role: LEADER
I20260921 18:43:47.996009 3302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d73670ab464974a69727e84e06425b" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 34233 } }
I20260921 18:43:47.996675 3297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:43:47.997087 3302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d73670ab464974a69727e84e06425b. Latest consensus state: current_term: 1 leader_uuid: "66d73670ab464974a69727e84e06425b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d73670ab464974a69727e84e06425b" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 34233 } } }
I20260921 18:43:47.997179 3302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [sys.catalog]: This master's current role is: LEADER
I20260921 18:43:47.997552 3306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:43:47.999679 3303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d73670ab464974a69727e84e06425b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d73670ab464974a69727e84e06425b" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 34233 } } }
I20260921 18:43:47.999789 3303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b [sys.catalog]: This master's current role is: LEADER
I20260921 18:43:48.000589 3306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:43:48.006378 3306 catalog_manager.cc:1347] Generated new cluster ID: ad9d098841ed4915a2267b3948edf09b
I20260921 18:43:48.006438 3306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:43:48.020859 3306 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:43:48.022013 3306 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:43:48.034607 3306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d73670ab464974a69727e84e06425b: Generated new TSK 0
I20260921 18:43:48.035380 3306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:43:48.042712 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--builtin_ntp_servers=127.3.36.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260921 18:43:48.149704 3321 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:43:48.149956 3321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:43:48.149994 3321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:43:48.150024 3321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:43:48.153877 3321 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260921 18:43:48.153932 3321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:43:48.153993 3321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:43:48.158468 3321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--never_fsync=true
--heap_profile_path=/tmp/kudu.3321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:43:48.159294 3321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:43:48.160454 3321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:43:48.168843 3326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:48.168897 3327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:48.168892 3329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:43:48.169144 3321 server_base.cc:1034] running on GCE node
I20260921 18:43:48.169641 3321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:43:48.170217 3321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:43:48.171401 3321 hybrid_clock.cc:648] HybridClock initialized: now 1790016228171394 us; error 29 us; skew 500 ppm
I20260921 18:43:48.173048 3321 webserver.cc:460] Webserver started at http://127.3.36.129:35703/ using document root <none> and password file <none>
I20260921 18:43:48.173697 3321 fs_manager.cc:359] Metadata directory not provided
I20260921 18:43:48.173796 3321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:43:48.174064 3321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:43:48.176023 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "cc0a0d7fa33a402191e854b4f99fd110"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.176566 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "cc0a0d7fa33a402191e854b4f99fd110"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.180034 3321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260921 18:43:48.182261 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.183344 3321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:43:48.183473 3321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "cc0a0d7fa33a402191e854b4f99fd110"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.183570 3321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:43:48.205451 3321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:43:48.206086 3321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:43:48.206267 3321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:43:48.206801 3321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:43:48.207647 3321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:43:48.207727 3321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.207792 3321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:43:48.207844 3321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.216145 3321 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:37617
I20260921 18:43:48.216169 3448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:37617 every 8 connection(s)
I20260921 18:43:48.216948 3321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20260921 18:43:48.218838 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3321
I20260921 18:43:48.218959 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20260921 18:43:48.221387 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--builtin_ntp_servers=127.3.36.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260921 18:43:48.228820 3449 heartbeater.cc:347] Connected to a master server at 127.3.36.190:34233
I20260921 18:43:48.229066 3449 heartbeater.cc:464] Registering TS with master...
I20260921 18:43:48.229691 3449 heartbeater.cc:511] Master 127.3.36.190:34233 requested a full tablet report, sending...
I20260921 18:43:48.231206 3261 ts_manager.cc:194] Registered new tserver with Master: cc0a0d7fa33a402191e854b4f99fd110 (127.3.36.129:37617)
I20260921 18:43:48.232268 3261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:47171
W20260921 18:43:48.328701 3452 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:43:48.328934 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:43:48.328980 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:43:48.329033 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:43:48.332861 3452 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260921 18:43:48.332913 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:43:48.333025 3452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:43:48.337571 3452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--never_fsync=true
--heap_profile_path=/tmp/kudu.3452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:43:48.338472 3452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:43:48.339694 3452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:43:48.348189 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:43:48.348304 3452 server_base.cc:1034] running on GCE node
W20260921 18:43:48.348242 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:48.348242 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:43:48.348886 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:43:48.349460 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:43:48.350684 3452 hybrid_clock.cc:648] HybridClock initialized: now 1790016228350672 us; error 23 us; skew 500 ppm
I20260921 18:43:48.352370 3452 webserver.cc:460] Webserver started at http://127.3.36.130:32999/ using document root <none> and password file <none>
I20260921 18:43:48.352900 3452 fs_manager.cc:359] Metadata directory not provided
I20260921 18:43:48.352986 3452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:43:48.353201 3452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:43:48.354946 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "c131bf791cec4489846467e0ace5b3d4"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.355461 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "c131bf791cec4489846467e0ace5b3d4"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.358836 3452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260921 18:43:48.360886 3466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.362023 3452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:43:48.362149 3452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "c131bf791cec4489846467e0ace5b3d4"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.362248 3452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:43:48.380507 3452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:43:48.381063 3452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:43:48.381235 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:43:48.381798 3452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:43:48.382668 3452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:43:48.382735 3452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.382797 3452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:43:48.382848 3452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.391606 3452 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:46097
I20260921 18:43:48.391630 3579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:46097 every 8 connection(s)
I20260921 18:43:48.392385 3452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20260921 18:43:48.397768 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3452
I20260921 18:43:48.397873 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20260921 18:43:48.399948 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--builtin_ntp_servers=127.3.36.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20260921 18:43:48.402711 3580 heartbeater.cc:347] Connected to a master server at 127.3.36.190:34233
I20260921 18:43:48.402921 3580 heartbeater.cc:464] Registering TS with master...
I20260921 18:43:48.403360 3580 heartbeater.cc:511] Master 127.3.36.190:34233 requested a full tablet report, sending...
I20260921 18:43:48.404393 3261 ts_manager.cc:194] Registered new tserver with Master: c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097)
I20260921 18:43:48.405244 3261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:43515
W20260921 18:43:48.505599 3583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:43:48.505820 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:43:48.505838 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:43:48.505867 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:43:48.509732 3583 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260921 18:43:48.509783 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:43:48.509832 3583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:43:48.514293 3583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--never_fsync=true
--heap_profile_path=/tmp/kudu.3583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:43:48.515096 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:43:48.516255 3583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260921 18:43:48.524670 3583 server_base.cc:1034] running on GCE node
W20260921 18:43:48.524593 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:48.524583 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:48.524575 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:43:48.525266 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:43:48.525861 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:43:48.527036 3583 hybrid_clock.cc:648] HybridClock initialized: now 1790016228527019 us; error 25 us; skew 500 ppm
I20260921 18:43:48.528777 3583 webserver.cc:460] Webserver started at http://127.3.36.131:38289/ using document root <none> and password file <none>
I20260921 18:43:48.529296 3583 fs_manager.cc:359] Metadata directory not provided
I20260921 18:43:48.529354 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:43:48.529629 3583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:43:48.531364 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "8f9777c110084551b8104d292349b027"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.531883 3583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "8f9777c110084551b8104d292349b027"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.535322 3583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:43:48.537460 3597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.538618 3583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:43:48.538728 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "8f9777c110084551b8104d292349b027"
format_stamp: "Formatted at 2026-09-21 18:43:48 on dist-test-slave-l4st"
I20260921 18:43:48.538816 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:43:48.570402 3583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:43:48.571069 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:43:48.571254 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:43:48.571822 3583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:43:48.572719 3583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:43:48.572763 3583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.572825 3583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:43:48.572865 3583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:48.581370 3583 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:46513
I20260921 18:43:48.581401 3710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:46513 every 8 connection(s)
I20260921 18:43:48.582227 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20260921 18:43:48.586009 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3583
I20260921 18:43:48.586112 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20260921 18:43:48.592003 3711 heartbeater.cc:347] Connected to a master server at 127.3.36.190:34233
I20260921 18:43:48.592206 3711 heartbeater.cc:464] Registering TS with master...
I20260921 18:43:48.592638 3711 heartbeater.cc:511] Master 127.3.36.190:34233 requested a full tablet report, sending...
I20260921 18:43:48.593559 3261 ts_manager.cc:194] Registered new tserver with Master: 8f9777c110084551b8104d292349b027 (127.3.36.131:46513)
I20260921 18:43:48.594074 3261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:39909
I20260921 18:43:48.600189 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:43:48.601985 3218 test_util.cc:268] Using random seed: -1586319234
I20260921 18:43:48.614847 3261 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49752:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260921 18:43:48.615792 3261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:43:48.631421 3514 tablet_service.cc:1467] Processing CreateTablet for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 (DEFAULT_TABLE table=test-workload [id=baea761fd5e44c5fa56a372446251215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:43:48.631680 3645 tablet_service.cc:1467] Processing CreateTablet for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 (DEFAULT_TABLE table=test-workload [id=baea761fd5e44c5fa56a372446251215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:43:48.631909 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 (DEFAULT_TABLE table=test-workload [id=baea761fd5e44c5fa56a372446251215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:43:48.632477 3514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:43:48.632581 3645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:43:48.632745 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:43:48.637662 3728 tablet_bootstrap.cc:492] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027: Bootstrap starting.
I20260921 18:43:48.639156 3730 tablet_bootstrap.cc:492] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: Bootstrap starting.
I20260921 18:43:48.640625 3728 tablet_bootstrap.cc:654] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027: Neither blocks nor log segments found. Creating new log.
I20260921 18:43:48.641137 3730 tablet_bootstrap.cc:654] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: Neither blocks nor log segments found. Creating new log.
I20260921 18:43:48.641319 3728 log.cc:824] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027: Log is configured to *not* fsync() on all Append() calls
I20260921 18:43:48.641739 3729 tablet_bootstrap.cc:492] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: Bootstrap starting.
I20260921 18:43:48.641916 3730 log.cc:824] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: Log is configured to *not* fsync() on all Append() calls
I20260921 18:43:48.643236 3728 tablet_bootstrap.cc:492] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027: No bootstrap required, opened a new log
I20260921 18:43:48.643450 3728 ts_tablet_manager.cc:1397] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260921 18:43:48.643500 3729 tablet_bootstrap.cc:654] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: Neither blocks nor log segments found. Creating new log.
I20260921 18:43:48.644264 3729 log.cc:824] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: Log is configured to *not* fsync() on all Append() calls
I20260921 18:43:48.645903 3729 tablet_bootstrap.cc:492] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: No bootstrap required, opened a new log
I20260921 18:43:48.646080 3729 ts_tablet_manager.cc:1397] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260921 18:43:48.646677 3730 tablet_bootstrap.cc:492] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: No bootstrap required, opened a new log
I20260921 18:43:48.646835 3730 ts_tablet_manager.cc:1397] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260921 18:43:48.647135 3728 raft_consensus.cc:348] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.647351 3728 raft_consensus.cc:729] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9777c110084551b8104d292349b027, State: Initialized, Role: FOLLOWER
I20260921 18:43:48.647624 3728 consensus_queue.cc:260] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.648211 3711 heartbeater.cc:503] Master 127.3.36.190:34233 was elected leader, sending a full tablet report...
I20260921 18:43:48.648547 3728 ts_tablet_manager.cc:1428] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:43:48.650290 3729 raft_consensus.cc:348] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.650523 3729 raft_consensus.cc:729] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0a0d7fa33a402191e854b4f99fd110, State: Initialized, Role: FOLLOWER
I20260921 18:43:48.650830 3729 consensus_queue.cc:260] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.650961 3730 raft_consensus.cc:348] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.651223 3730 raft_consensus.cc:729] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c131bf791cec4489846467e0ace5b3d4, State: Initialized, Role: FOLLOWER
I20260921 18:43:48.651481 3449 heartbeater.cc:503] Master 127.3.36.190:34233 was elected leader, sending a full tablet report...
I20260921 18:43:48.651504 3730 consensus_queue.cc:260] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.652088 3580 heartbeater.cc:503] Master 127.3.36.190:34233 was elected leader, sending a full tablet report...
I20260921 18:43:48.652371 3730 ts_tablet_manager.cc:1428] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:43:48.653249 3729 ts_tablet_manager.cc:1428] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260921 18:43:48.684646 3574 generic_service.cc:139] {username='slave'} at 127.0.0.1:42186 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260921 18:43:48.684777 3574 generic_service.cc:175] {username='slave'} at 127.0.0.1:42186 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.3.36.130:46097'
W20260921 18:43:48.685647 3574 generic_service.cc:139] {username='slave'} at 127.0.0.1:42186 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:48.685722 3574 generic_service.cc:175] {username='slave'} at 127.0.0.1:42186 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260921 18:43:48.689589 3705 generic_service.cc:139] {username='slave'} at 127.0.0.1:41360 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260921 18:43:48.689704 3705 generic_service.cc:175] {username='slave'} at 127.0.0.1:41360 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.3.36.130:46097'
W20260921 18:43:48.690389 3705 generic_service.cc:139] {username='slave'} at 127.0.0.1:41360 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:48.690460 3705 generic_service.cc:175] {username='slave'} at 127.0.0.1:41360 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260921 18:43:48.694029 3443 generic_service.cc:139] {username='slave'} at 127.0.0.1:39406 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20260921 18:43:48.694146 3443 generic_service.cc:175] {username='slave'} at 127.0.0.1:39406 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.3.36.130:46097'
W20260921 18:43:48.694872 3443 generic_service.cc:139] {username='slave'} at 127.0.0.1:39406 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:48.694943 3443 generic_service.cc:175] {username='slave'} at 127.0.0.1:39406 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260921 18:43:48.695689 3665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67fb3d4fc7c34933b11fb6c6b1f16e17"
dest_uuid: "8f9777c110084551b8104d292349b027"
from {username='slave'} at 127.0.0.1:41346
I20260921 18:43:48.695879 3665 raft_consensus.cc:482] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:43:48.696003 3665 raft_consensus.cc:3037] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:43:48.697037 3665 raft_consensus.cc:504] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.698303 3665 leader_election.cc:288] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [CANDIDATE]: Term 1 election: Requested vote from peers c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097), cc0a0d7fa33a402191e854b4f99fd110 (127.3.36.129:37617)
W20260921 18:43:48.698565 3739 leader_election.cc:334] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097
I20260921 18:43:48.701867 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb3d4fc7c34933b11fb6c6b1f16e17" candidate_uuid: "8f9777c110084551b8104d292349b027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cc0a0d7fa33a402191e854b4f99fd110"
I20260921 18:43:48.702100 3403 raft_consensus.cc:3037] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:43:48.703017 3403 raft_consensus.cc:2454] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9777c110084551b8104d292349b027 in term 1.
I20260921 18:43:48.703332 3600 leader_election.cc:302] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f9777c110084551b8104d292349b027, cc0a0d7fa33a402191e854b4f99fd110; no voters: c131bf791cec4489846467e0ace5b3d4
I20260921 18:43:48.703608 3734 raft_consensus.cc:2781] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:43:48.703822 3734 raft_consensus.cc:686] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9777c110084551b8104d292349b027, State: Running, Role: LEADER
I20260921 18:43:48.704154 3734 consensus_queue.cc:237] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:48.707756 3261 catalog_manager.cc:5495] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9777c110084551b8104d292349b027 (127.3.36.131). New cstate: current_term: 1 leader_uuid: "8f9777c110084551b8104d292349b027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } health_report { overall_health: UNKNOWN } } }
W20260921 18:43:48.717936 3450 tablet.cc:2369] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:43:48.785959 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 1: this message will repeat every 5th retry.
I20260921 18:43:48.786163 3734 consensus_queue.cc:1035] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260921 18:43:48.893404 3581 tablet.cc:2369] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:43:49.236272 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 6: this message will repeat every 5th retry.
W20260921 18:43:49.766743 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 11: this message will repeat every 5th retry.
W20260921 18:43:50.282636 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 16: this message will repeat every 5th retry.
W20260921 18:43:50.763002 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 21: this message will repeat every 5th retry.
W20260921 18:43:51.275470 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 26: this message will repeat every 5th retry.
I20260921 18:43:51.743337 3734 consensus_queue.cc:579] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Leader has been unable to successfully communicate with peer c131bf791cec4489846467e0ace5b3d4 for more than 3 seconds (3.039s)
W20260921 18:43:51.743842 3739 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 31: this message will repeat every 5th retry.
W20260921 18:43:51.745617 3260 catalog_manager.cc:5173] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 with cas_config_opid_index -1: no extra replica candidate found for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 (table test-workload [id=baea761fd5e44c5fa56a372446251215]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20260921 18:43:51.768328 3574 generic_service.cc:139] {username='slave'} at 127.0.0.1:42186 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:51.768428 3574 generic_service.cc:175] {username='slave'} at 127.0.0.1:42186 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260921 18:43:51.769176 3705 generic_service.cc:139] {username='slave'} at 127.0.0.1:41360 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:51.769275 3705 generic_service.cc:175] {username='slave'} at 127.0.0.1:41360 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20260921 18:43:51.770042 3443 generic_service.cc:139] {username='slave'} at 127.0.0.1:39406 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:51.770117 3443 generic_service.cc:175] {username='slave'} at 127.0.0.1:39406 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20260921 18:43:51.827528 3534 raft_consensus.cc:3037] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:43:51.920739 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.132:0
--local_ip_for_outbound_sockets=127.3.36.132
--webserver_interface=127.3.36.132
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--builtin_ntp_servers=127.3.36.148:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20260921 18:43:52.026871 3752 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:43:52.027163 3752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:43:52.027209 3752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:43:52.027273 3752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:43:52.031452 3752 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20260921 18:43:52.031543 3752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:43:52.031629 3752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.132
I20260921 18:43:52.036464 3752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.36.132
--webserver_port=0
--tserver_master_addrs=127.3.36.190:34233
--never_fsync=true
--heap_profile_path=/tmp/kudu.3752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.3.36.132
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:43:52.037338 3752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:43:52.038614 3752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:43:52.047350 3758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:52.047315 3757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:43:52.047315 3760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:43:52.047386 3752 server_base.cc:1034] running on GCE node
I20260921 18:43:52.047909 3752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:43:52.048473 3752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:43:52.049625 3752 hybrid_clock.cc:648] HybridClock initialized: now 1790016232049596 us; error 41 us; skew 500 ppm
I20260921 18:43:52.059237 3752 webserver.cc:460] Webserver started at http://127.3.36.132:33621/ using document root <none> and password file <none>
I20260921 18:43:52.059733 3752 fs_manager.cc:359] Metadata directory not provided
I20260921 18:43:52.059787 3752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:43:52.059950 3752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:43:52.061754 3752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "ab93b2294d7e445d81c657db6df7d32f"
format_stamp: "Formatted at 2026-09-21 18:43:52 on dist-test-slave-l4st"
I20260921 18:43:52.062248 3752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "ab93b2294d7e445d81c657db6df7d32f"
format_stamp: "Formatted at 2026-09-21 18:43:52 on dist-test-slave-l4st"
I20260921 18:43:52.065683 3752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260921 18:43:52.068011 3766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:52.069108 3752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:43:52.069192 3752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "ab93b2294d7e445d81c657db6df7d32f"
format_stamp: "Formatted at 2026-09-21 18:43:52 on dist-test-slave-l4st"
I20260921 18:43:52.069262 3752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:43:52.093253 3752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:43:52.093858 3752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:43:52.094002 3752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:43:52.094496 3752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:43:52.095315 3752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:43:52.095357 3752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:52.095395 3752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:43:52.095412 3752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:43:52.104050 3752 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.132:44455
I20260921 18:43:52.104102 3879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.132:44455 every 8 connection(s)
I20260921 18:43:52.104840 3752 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20260921 18:43:52.106541 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3752
I20260921 18:43:52.106647 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20260921 18:43:52.114785 3880 heartbeater.cc:347] Connected to a master server at 127.3.36.190:34233
I20260921 18:43:52.115020 3880 heartbeater.cc:464] Registering TS with master...
I20260921 18:43:52.115460 3880 heartbeater.cc:511] Master 127.3.36.190:34233 requested a full tablet report, sending...
I20260921 18:43:52.116392 3260 ts_manager.cc:194] Registered new tserver with Master: ab93b2294d7e445d81c657db6df7d32f (127.3.36.132:44455)
I20260921 18:43:52.116940 3260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.132:59791
I20260921 18:43:52.118624 3665 consensus_queue.cc:237] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } peers { permanent_uuid: "ab93b2294d7e445d81c657db6df7d32f" member_type: NON_VOTER last_known_addr { host: "127.3.36.132" port: 44455 } attrs { promote: true } }
I20260921 18:43:52.120410 3403 raft_consensus.cc:1261] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 1 FOLLOWER]: Refusing update from remote peer 8f9777c110084551b8104d292349b027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:43:52.120419 3534 raft_consensus.cc:1261] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 1 FOLLOWER]: Refusing update from remote peer 8f9777c110084551b8104d292349b027: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:43:52.120848 3746 consensus_queue.cc:1035] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:43:52.121068 3745 consensus_queue.cc:1035] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:43:52.122831 3745 raft_consensus.cc:2932] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } peers { permanent_uuid: "ab93b2294d7e445d81c657db6df7d32f" member_type: NON_VOTER last_known_addr { host: "127.3.36.132" port: 44455 } attrs { promote: true } } }
I20260921 18:43:52.124928 3246 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20260921 18:43:52.125172 3403 raft_consensus.cc:2932] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } peers { permanent_uuid: "ab93b2294d7e445d81c657db6df7d32f" member_type: NON_VOTER last_known_addr { host: "127.3.36.132" port: 44455 } attrs { promote: true } } }
I20260921 18:43:52.125571 3260 catalog_manager.cc:5495] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 reported cstate change: config changed from index -1 to 2, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) added. New cstate: current_term: 1 leader_uuid: "8f9777c110084551b8104d292349b027" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab93b2294d7e445d81c657db6df7d32f" member_type: NON_VOTER last_known_addr { host: "127.3.36.132" port: 44455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260921 18:43:52.126298 3601 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer ab93b2294d7e445d81c657db6df7d32f (127.3.36.132:44455): Couldn't send request to peer ab93b2294d7e445d81c657db6df7d32f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67fb3d4fc7c34933b11fb6c6b1f16e17. This is attempt 1: this message will repeat every 5th retry.
I20260921 18:43:52.126619 3534 raft_consensus.cc:2932] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } peers { permanent_uuid: "ab93b2294d7e445d81c657db6df7d32f" member_type: NON_VOTER last_known_addr { host: "127.3.36.132" port: 44455 } attrs { promote: true } } }
I20260921 18:43:52.129063 3665 consensus_queue.cc:237] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } }
I20260921 18:43:52.130116 3403 raft_consensus.cc:1261] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 1 FOLLOWER]: Refusing update from remote peer 8f9777c110084551b8104d292349b027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260921 18:43:52.130267 3534 raft_consensus.cc:1261] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 1 FOLLOWER]: Refusing update from remote peer 8f9777c110084551b8104d292349b027: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260921 18:43:52.130430 3746 consensus_queue.cc:1035] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260921 18:43:52.130604 3734 consensus_queue.cc:1035] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260921 18:43:52.131893 3745 raft_consensus.cc:2932] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } }
I20260921 18:43:52.132293 3403 raft_consensus.cc:2932] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P cc0a0d7fa33a402191e854b4f99fd110 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } }
I20260921 18:43:52.132360 3534 raft_consensus.cc:2932] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P c131bf791cec4489846467e0ace5b3d4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } } }
I20260921 18:43:52.134023 3246 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260921 18:43:52.134433 3261 catalog_manager.cc:5495] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 reported cstate change: config changed from index 2 to 3, NON_VOTER ab93b2294d7e445d81c657db6df7d32f (127.3.36.132) evicted. New cstate: current_term: 1 leader_uuid: "8f9777c110084551b8104d292349b027" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c131bf791cec4489846467e0ace5b3d4" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f9777c110084551b8104d292349b027" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 46513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc0a0d7fa33a402191e854b4f99fd110" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37617 } health_report { overall_health: HEALTHY } } }
I20260921 18:43:52.140334 3814 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 with delete_type TABLET_DATA_TOMBSTONED (TS ab93b2294d7e445d81c657db6df7d32f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:33656
W20260921 18:43:52.141597 3248 catalog_manager.cc:4808] TS ab93b2294d7e445d81c657db6df7d32f (127.3.36.132:44455): delete failed for tablet 67fb3d4fc7c34933b11fb6c6b1f16e17 because the tablet was not found. No further retry: Not found: Tablet not found: 67fb3d4fc7c34933b11fb6c6b1f16e17
W20260921 18:43:52.159669 3574 generic_service.cc:139] {username='slave'} at 127.0.0.1:42186 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:52.159735 3574 generic_service.cc:175] {username='slave'} at 127.0.0.1:42186 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260921 18:43:52.160454 3705 generic_service.cc:139] {username='slave'} at 127.0.0.1:41360 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:52.160513 3705 generic_service.cc:175] {username='slave'} at 127.0.0.1:41360 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20260921 18:43:52.161227 3443 generic_service.cc:139] {username='slave'} at 127.0.0.1:39406 forcing change of non-runtime-safe flag fail_dns_resolution
I20260921 18:43:52.161302 3443 generic_service.cc:175] {username='slave'} at 127.0.0.1:39406 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20260921 18:43:52.162200 3665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67fb3d4fc7c34933b11fb6c6b1f16e17"
dest_uuid: "8f9777c110084551b8104d292349b027"
mode: GRACEFUL
new_leader_uuid: "8f9777c110084551b8104d292349b027"
from {username='slave'} at 127.0.0.1:41346
I20260921 18:43:52.162395 3665 raft_consensus.cc:595] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 [term 1 LEADER]: Received request to transfer leadership to TS 8f9777c110084551b8104d292349b027
I20260921 18:43:53.118789 3880 heartbeater.cc:503] Master 127.3.36.190:34233 was elected leader, sending a full tablet report...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus_election-itest.cc:292: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
I20260921 18:44:22.233413 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3321
W20260921 18:44:22.238987 3600 connection.cc:535] client connection to 127.3.36.129:37617 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260921 18:44:22.239286 3600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:44:22.239471 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3452
W20260921 18:44:22.241436 3600 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer cc0a0d7fa33a402191e854b4f99fd110 (127.3.36.129:37617): Couldn't send request to peer cc0a0d7fa33a402191e854b4f99fd110. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.129:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260921 18:44:22.247043 3598 connection.cc:535] client connection to 127.3.36.130:46097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260921 18:44:22.247430 3897 consensus_peers.cc:487] T 67fb3d4fc7c34933b11fb6c6b1f16e17 P 8f9777c110084551b8104d292349b027 -> Peer c131bf791cec4489846467e0ace5b3d4 (127.3.36.130:46097): Couldn't send request to peer c131bf791cec4489846467e0ace5b3d4. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.3.36.130:46097. This is attempt 1: this message will repeat every 5th retry.
I20260921 18:44:22.247551 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3583
I20260921 18:44:22.252993 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3752
I20260921 18:44:22.258131 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3230
2026-09-21T18:44:22Z chronyd exiting
I20260921 18:44:22.275307 3218 test_util.cc:175] -----------------------------------------------
I20260921 18:44:22.275360 3218 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1790016227753869-3218-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (34498 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20260921 18:44:22.276113 3218 test_util.cc:268] Using random seed: -1552645108
I20260921 18:44:22.276867 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:44:22.276903 3218 ts_itest-base.cc:113] --------------
I20260921 18:44:22.276921 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:44:22.276958 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:44:22.276978 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:44:22Z Disabled control of system clock
I20260921 18:44:22.285320 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:37511
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:37511 with env {}
W20260921 18:44:22.389770 3906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:22.390030 3906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:22.390100 3906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:22.393937 3906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:44:22.394008 3906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:22.394042 3906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:44:22.394090 3906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:44:22.398700 3906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:37511
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:37511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:22.399621 3906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:22.400861 3906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:22.408617 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.408603 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.408591 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:22.408977 3906 server_base.cc:1034] running on GCE node
I20260921 18:44:22.409526 3906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:22.410542 3906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:22.411717 3906 hybrid_clock.cc:648] HybridClock initialized: now 1790016262411705 us; error 26 us; skew 500 ppm
I20260921 18:44:22.417272 3906 webserver.cc:460] Webserver started at http://127.3.36.190:46035/ using document root <none> and password file <none>
I20260921 18:44:22.417858 3906 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:22.417972 3906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:22.418175 3906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:22.419926 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "561543b86f6840a2ba43e1c5ff8503d5"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.420466 3906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "561543b86f6840a2ba43e1c5ff8503d5"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.423835 3906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:44:22.426011 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.427089 3906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:22.427217 3906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "561543b86f6840a2ba43e1c5ff8503d5"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.427318 3906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:22.438081 3906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:22.438622 3906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:22.438792 3906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:22.445088 3906 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:37511
I20260921 18:44:22.445096 3972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:37511 every 8 connection(s)
I20260921 18:44:22.445950 3906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:44:22.448846 3973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:22.450054 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3906
I20260921 18:44:22.450166 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:44:22.454319 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5: Bootstrap starting.
I20260921 18:44:22.456761 3973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:22.457616 3973 log.cc:824] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:22.459328 3973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5: No bootstrap required, opened a new log
I20260921 18:44:22.462733 3973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561543b86f6840a2ba43e1c5ff8503d5" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37511 } }
I20260921 18:44:22.462925 3973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:22.462958 3973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561543b86f6840a2ba43e1c5ff8503d5, State: Initialized, Role: FOLLOWER
I20260921 18:44:22.463313 3973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561543b86f6840a2ba43e1c5ff8503d5" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37511 } }
I20260921 18:44:22.463423 3973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:44:22.463467 3973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:44:22.463539 3973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:22.464196 3973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561543b86f6840a2ba43e1c5ff8503d5" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37511 } }
I20260921 18:44:22.464496 3973 leader_election.cc:302] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 561543b86f6840a2ba43e1c5ff8503d5; no voters:
I20260921 18:44:22.464735 3973 leader_election.cc:288] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:44:22.464845 3978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:44:22.465085 3978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [term 1 LEADER]: Becoming Leader. State: Replica: 561543b86f6840a2ba43e1c5ff8503d5, State: Running, Role: LEADER
I20260921 18:44:22.465413 3978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561543b86f6840a2ba43e1c5ff8503d5" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37511 } }
I20260921 18:44:22.465916 3973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:44:22.467034 3980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 561543b86f6840a2ba43e1c5ff8503d5. Latest consensus state: current_term: 1 leader_uuid: "561543b86f6840a2ba43e1c5ff8503d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561543b86f6840a2ba43e1c5ff8503d5" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37511 } } }
I20260921 18:44:22.467160 3980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [sys.catalog]: This master's current role is: LEADER
I20260921 18:44:22.467424 3979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "561543b86f6840a2ba43e1c5ff8503d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561543b86f6840a2ba43e1c5ff8503d5" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37511 } } }
I20260921 18:44:22.467486 3987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:44:22.467577 3979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5 [sys.catalog]: This master's current role is: LEADER
I20260921 18:44:22.470379 3987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:44:22.475373 3987 catalog_manager.cc:1347] Generated new cluster ID: 5dc76a07cd1d47dc9830e030ef780c70
I20260921 18:44:22.475432 3987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:44:22.486115 3987 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:44:22.487147 3987 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:44:22.498519 3987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 561543b86f6840a2ba43e1c5ff8503d5: Generated new TSK 0
I20260921 18:44:22.499193 3987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:44:22.504931 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37511
--builtin_ntp_servers=127.3.36.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260921 18:44:22.610041 3997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:22.610262 3997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:22.610322 3997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:22.614342 3997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:22.614441 3997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:44:22.619024 3997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37511
--never_fsync=true
--heap_profile_path=/tmp/kudu.3997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:22.619958 3997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:22.621645 3997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:22.630895 4003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.630975 4005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.630914 4002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:22.631251 3997 server_base.cc:1034] running on GCE node
I20260921 18:44:22.631618 3997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:22.632241 3997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:22.633443 3997 hybrid_clock.cc:648] HybridClock initialized: now 1790016262633429 us; error 29 us; skew 500 ppm
I20260921 18:44:22.635164 3997 webserver.cc:460] Webserver started at http://127.3.36.129:44917/ using document root <none> and password file <none>
I20260921 18:44:22.635694 3997 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:22.635777 3997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:22.635994 3997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:22.637809 3997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1af3525fb39a457a926165e1bb12608c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.638334 3997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1af3525fb39a457a926165e1bb12608c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.641865 3997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260921 18:44:22.644192 4011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.645313 3997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:22.645431 3997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1af3525fb39a457a926165e1bb12608c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.645576 3997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:22.656380 3997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:22.656934 3997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:22.657106 3997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:22.657637 3997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:22.658449 3997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:22.658517 3997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.658579 3997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:22.658625 3997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.666676 3997 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:42261
I20260921 18:44:22.666698 4124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:42261 every 8 connection(s)
I20260921 18:44:22.667558 3997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:44:22.670549 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 3997
I20260921 18:44:22.670660 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:44:22.672932 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37511
--builtin_ntp_servers=127.3.36.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20260921 18:44:22.678297 4125 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37511
I20260921 18:44:22.678551 4125 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:22.679004 4125 heartbeater.cc:511] Master 127.3.36.190:37511 requested a full tablet report, sending...
I20260921 18:44:22.680368 3937 ts_manager.cc:194] Registered new tserver with Master: 1af3525fb39a457a926165e1bb12608c (127.3.36.129:42261)
I20260921 18:44:22.681425 3937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:41147
W20260921 18:44:22.779215 4128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:22.779454 4128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:22.779520 4128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:22.783504 4128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:22.783646 4128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:44:22.788188 4128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37511
--never_fsync=true
--heap_profile_path=/tmp/kudu.4128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:22.789161 4128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:22.790472 4128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:22.799351 4136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.799379 4133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.799362 4134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:22.799682 4128 server_base.cc:1034] running on GCE node
I20260921 18:44:22.800099 4128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:22.800673 4128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:22.801839 4128 hybrid_clock.cc:648] HybridClock initialized: now 1790016262801800 us; error 54 us; skew 500 ppm
I20260921 18:44:22.803571 4128 webserver.cc:460] Webserver started at http://127.3.36.130:37145/ using document root <none> and password file <none>
I20260921 18:44:22.804095 4128 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:22.804181 4128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:22.804396 4128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:22.806262 4128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "06248013c41c42e99a2e41ab3dfa336c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.806787 4128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "06248013c41c42e99a2e41ab3dfa336c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.810222 4128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260921 18:44:22.812832 4142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.813949 4128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:22.814069 4128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "06248013c41c42e99a2e41ab3dfa336c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.814165 4128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:22.828351 4128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:22.828962 4128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:22.829126 4128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:22.829641 4128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:22.830547 4128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:22.830607 4128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.830677 4128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:22.830721 4128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.839039 4128 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:37307
I20260921 18:44:22.839052 4255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:37307 every 8 connection(s)
I20260921 18:44:22.839833 4128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:44:22.849064 4256 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37511
I20260921 18:44:22.849313 4256 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:22.849320 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 4128
I20260921 18:44:22.849416 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:44:22.849897 4256 heartbeater.cc:511] Master 127.3.36.190:37511 requested a full tablet report, sending...
I20260921 18:44:22.850881 3936 ts_manager.cc:194] Registered new tserver with Master: 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307)
I20260921 18:44:22.851445 3936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:58595
I20260921 18:44:22.851523 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37511
--builtin_ntp_servers=127.3.36.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260921 18:44:22.960019 4259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:22.960235 4259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:22.960294 4259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:22.964232 4259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:22.964334 4259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:44:22.969195 4259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37511
--never_fsync=true
--heap_profile_path=/tmp/kudu.4259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:22.970144 4259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:22.971346 4259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:22.980065 4264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.980116 4267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:22.980093 4265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:22.980265 4259 server_base.cc:1034] running on GCE node
I20260921 18:44:22.980659 4259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:22.981258 4259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:22.982398 4259 hybrid_clock.cc:648] HybridClock initialized: now 1790016262982387 us; error 56 us; skew 500 ppm
I20260921 18:44:22.984028 4259 webserver.cc:460] Webserver started at http://127.3.36.131:43339/ using document root <none> and password file <none>
I20260921 18:44:22.984537 4259 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:22.984596 4259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:22.984817 4259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:22.986604 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fc8fd3f01f1d425692d1f4a406422256"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.987108 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fc8fd3f01f1d425692d1f4a406422256"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.990481 4259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260921 18:44:22.992645 4273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:22.993739 4259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:22.993850 4259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fc8fd3f01f1d425692d1f4a406422256"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:22.993943 4259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:23.003752 4259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:23.004315 4259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:23.004477 4259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:23.004946 4259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:23.005867 4259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:23.005939 4259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:23.006021 4259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:23.006065 4259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:23.014068 4259 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:34861
I20260921 18:44:23.014097 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:34861 every 8 connection(s)
I20260921 18:44:23.014832 4259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:44:23.016839 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 4259
I20260921 18:44:23.016923 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:44:23.024636 4387 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37511
I20260921 18:44:23.024812 4387 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:23.025234 4387 heartbeater.cc:511] Master 127.3.36.190:37511 requested a full tablet report, sending...
I20260921 18:44:23.026117 3936 ts_manager.cc:194] Registered new tserver with Master: fc8fd3f01f1d425692d1f4a406422256 (127.3.36.131:34861)
I20260921 18:44:23.026624 3936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:59267
I20260921 18:44:23.030602 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:44:23.038607 3936 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59466:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260921 18:44:23.041384 3936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:44:23.057000 4059 tablet_service.cc:1467] Processing CreateTablet for tablet 419f624f4da64d41860b748b926a5713 (DEFAULT_TABLE table=TestTable [id=a04e4d1b06524974b9993322c377c6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:23.057276 4321 tablet_service.cc:1467] Processing CreateTablet for tablet 419f624f4da64d41860b748b926a5713 (DEFAULT_TABLE table=TestTable [id=a04e4d1b06524974b9993322c377c6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:23.057626 4190 tablet_service.cc:1467] Processing CreateTablet for tablet 419f624f4da64d41860b748b926a5713 (DEFAULT_TABLE table=TestTable [id=a04e4d1b06524974b9993322c377c6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:23.058133 4321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419f624f4da64d41860b748b926a5713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:23.058131 4059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419f624f4da64d41860b748b926a5713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:23.058463 4190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419f624f4da64d41860b748b926a5713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:23.063795 4404 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Bootstrap starting.
I20260921 18:44:23.065218 4405 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: Bootstrap starting.
I20260921 18:44:23.065399 4406 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: Bootstrap starting.
I20260921 18:44:23.066566 4404 tablet_bootstrap.cc:654] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:23.067355 4405 tablet_bootstrap.cc:654] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:23.067365 4406 tablet_bootstrap.cc:654] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:23.067390 4404 log.cc:824] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:23.068224 4406 log.cc:824] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:23.068224 4405 log.cc:824] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:23.069855 4404 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: No bootstrap required, opened a new log
I20260921 18:44:23.070022 4405 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: No bootstrap required, opened a new log
I20260921 18:44:23.070200 4405 ts_tablet_manager.cc:1397] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:44:23.070369 4404 ts_tablet_manager.cc:1397] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260921 18:44:23.070443 4406 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: No bootstrap required, opened a new log
I20260921 18:44:23.070586 4406 ts_tablet_manager.cc:1397] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:44:23.073725 4404 raft_consensus.cc:348] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.073962 4404 raft_consensus.cc:374] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:23.074014 4404 raft_consensus.cc:729] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06248013c41c42e99a2e41ab3dfa336c, State: Initialized, Role: FOLLOWER
I20260921 18:44:23.074460 4404 consensus_queue.cc:260] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.074648 4405 raft_consensus.cc:348] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.074918 4405 raft_consensus.cc:374] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:23.074787 4406 raft_consensus.cc:348] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.074975 4405 raft_consensus.cc:729] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8fd3f01f1d425692d1f4a406422256, State: Initialized, Role: FOLLOWER
I20260921 18:44:23.074996 4406 raft_consensus.cc:374] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:23.075043 4406 raft_consensus.cc:729] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af3525fb39a457a926165e1bb12608c, State: Initialized, Role: FOLLOWER
I20260921 18:44:23.075299 4256 heartbeater.cc:503] Master 127.3.36.190:37511 was elected leader, sending a full tablet report...
I20260921 18:44:23.075429 4405 consensus_queue.cc:260] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.075429 4406 consensus_queue.cc:260] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.075546 4404 ts_tablet_manager.cc:1428] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:44:23.076339 4406 ts_tablet_manager.cc:1428] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:44:23.076339 4405 ts_tablet_manager.cc:1428] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:44:23.076396 4387 heartbeater.cc:503] Master 127.3.36.190:37511 was elected leader, sending a full tablet report...
I20260921 18:44:23.077627 4125 heartbeater.cc:503] Master 127.3.36.190:37511 was elected leader, sending a full tablet report...
W20260921 18:44:23.090641 4257 tablet.cc:2369] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:44:23.168642 4126 tablet.cc:2369] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:44:23.265712 4388 tablet.cc:2369] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:44:23.456238 4410 raft_consensus.cc:482] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:23.456436 4410 raft_consensus.cc:504] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.457533 4410 leader_election.cc:288] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc8fd3f01f1d425692d1f4a406422256 (127.3.36.131:34861), 1af3525fb39a457a926165e1bb12608c (127.3.36.129:42261)
I20260921 18:44:23.461859 4341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419f624f4da64d41860b748b926a5713" candidate_uuid: "06248013c41c42e99a2e41ab3dfa336c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8fd3f01f1d425692d1f4a406422256" is_pre_election: true
I20260921 18:44:23.462139 4341 raft_consensus.cc:2454] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06248013c41c42e99a2e41ab3dfa336c in term 0.
I20260921 18:44:23.462146 4079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419f624f4da64d41860b748b926a5713" candidate_uuid: "06248013c41c42e99a2e41ab3dfa336c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af3525fb39a457a926165e1bb12608c" is_pre_election: true
I20260921 18:44:23.462435 4079 raft_consensus.cc:2454] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06248013c41c42e99a2e41ab3dfa336c in term 0.
I20260921 18:44:23.462550 4143 leader_election.cc:302] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06248013c41c42e99a2e41ab3dfa336c, fc8fd3f01f1d425692d1f4a406422256; no voters:
I20260921 18:44:23.462819 4410 raft_consensus.cc:2781] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:44:23.462916 4410 raft_consensus.cc:482] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:44:23.462975 4410 raft_consensus.cc:3037] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:23.463850 4410 raft_consensus.cc:504] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.464175 4410 leader_election.cc:288] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [CANDIDATE]: Term 1 election: Requested vote from peers fc8fd3f01f1d425692d1f4a406422256 (127.3.36.131:34861), 1af3525fb39a457a926165e1bb12608c (127.3.36.129:42261)
I20260921 18:44:23.464495 4341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419f624f4da64d41860b748b926a5713" candidate_uuid: "06248013c41c42e99a2e41ab3dfa336c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8fd3f01f1d425692d1f4a406422256"
I20260921 18:44:23.464542 4079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419f624f4da64d41860b748b926a5713" candidate_uuid: "06248013c41c42e99a2e41ab3dfa336c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1af3525fb39a457a926165e1bb12608c"
I20260921 18:44:23.464620 4341 raft_consensus.cc:3037] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:23.464676 4079 raft_consensus.cc:3037] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:23.465610 4341 raft_consensus.cc:2454] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06248013c41c42e99a2e41ab3dfa336c in term 1.
I20260921 18:44:23.465627 4079 raft_consensus.cc:2454] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06248013c41c42e99a2e41ab3dfa336c in term 1.
I20260921 18:44:23.465869 4143 leader_election.cc:302] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06248013c41c42e99a2e41ab3dfa336c, 1af3525fb39a457a926165e1bb12608c; no voters:
I20260921 18:44:23.466069 4410 raft_consensus.cc:2781] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:44:23.466300 4410 raft_consensus.cc:686] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 1 LEADER]: Becoming Leader. State: Replica: 06248013c41c42e99a2e41ab3dfa336c, State: Running, Role: LEADER
I20260921 18:44:23.466609 4410 consensus_queue.cc:237] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.469753 3936 catalog_manager.cc:5495] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c reported cstate change: term changed from 0 to 1, leader changed from <none> to 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130). New cstate: current_term: 1 leader_uuid: "06248013c41c42e99a2e41ab3dfa336c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } health_report { overall_health: UNKNOWN } } }
I20260921 18:44:23.497458 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:44:23.499118 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1af3525fb39a457a926165e1bb12608c to finish bootstrapping
I20260921 18:44:23.504549 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 06248013c41c42e99a2e41ab3dfa336c to finish bootstrapping
I20260921 18:44:23.509547 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fc8fd3f01f1d425692d1f4a406422256 to finish bootstrapping
I20260921 18:44:23.532413 4079 raft_consensus.cc:1261] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 1 FOLLOWER]: Refusing update from remote peer 06248013c41c42e99a2e41ab3dfa336c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:44:23.532413 4341 raft_consensus.cc:1261] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 1 FOLLOWER]: Refusing update from remote peer 06248013c41c42e99a2e41ab3dfa336c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:44:23.533062 4415 consensus_queue.cc:1035] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:44:23.533272 4410 consensus_queue.cc:1035] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:44:23.538887 4419 mvcc.cc:205] Tried to move back new op lower bound from 7331906615421939712 to 7331906615162109952. Current Snapshot: MvccSnapshot[applied={T|T < 7331906615421939712}]
I20260921 18:44:23.542023 4421 mvcc.cc:205] Tried to move back new op lower bound from 7331906615421939712 to 7331906615162109952. Current Snapshot: MvccSnapshot[applied={T|T < 7331906615421939712}]
I20260921 18:44:23.543134 4420 mvcc.cc:205] Tried to move back new op lower bound from 7331906615421939712 to 7331906615162109952. Current Snapshot: MvccSnapshot[applied={T|T < 7331906615421939712}]
I20260921 18:44:23.632025 4190 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:44:23.632289 4321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:44:23.632352 4059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
561543b86f6840a2ba43e1c5ff8503d5 | 127.3.36.190:37511 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
06248013c41c42e99a2e41ab3dfa336c | 127.3.36.130:37307 | HEALTHY | <none> | 1 | 0
1af3525fb39a457a926165e1bb12608c | 127.3.36.129:42261 | HEALTHY | <none> | 0 | 0
fc8fd3f01f1d425692d1f4a406422256 | 127.3.36.131:34861 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:42261
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:37307
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:34861
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:42261
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:37307
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:34861
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260921 18:44:23.660079 3218 log_verifier.cc:126] Checking tablet 419f624f4da64d41860b748b926a5713
I20260921 18:44:23.686708 3218 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 419f624f4da64d41860b748b926a5713
I20260921 18:44:23.720978 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 4128
I20260921 18:44:23.726935 4341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "419f624f4da64d41860b748b926a5713"
dest_uuid: "fc8fd3f01f1d425692d1f4a406422256"
from {username='slave'} at 127.0.0.1:46474
I20260921 18:44:23.727188 4341 raft_consensus.cc:482] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:44:23.727258 4341 raft_consensus.cc:3037] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:44:23.728597 4341 raft_consensus.cc:504] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.729902 4341 leader_election.cc:288] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [CANDIDATE]: Term 2 election: Requested vote from peers 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307), 1af3525fb39a457a926165e1bb12608c (127.3.36.129:42261)
W20260921 18:44:23.731578 4274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.130:37307: connect: Connection refused (error 111)
W20260921 18:44:23.732798 4274 leader_election.cc:334] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307): Network error: Client connection negotiation failed: client connection to 127.3.36.130:37307: connect: Connection refused (error 111)
I20260921 18:44:23.735203 4079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419f624f4da64d41860b748b926a5713" candidate_uuid: "fc8fd3f01f1d425692d1f4a406422256" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "1af3525fb39a457a926165e1bb12608c"
I20260921 18:44:23.735313 4079 raft_consensus.cc:3037] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 1 FOLLOWER]: Advancing to term 2
W20260921 18:44:23.736068 4390 meta_cache.cc:302] tablet 419f624f4da64d41860b748b926a5713: replica 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307) has failed: Network error: Client connection negotiation failed: client connection to 127.3.36.130:37307: connect: Connection refused (error 111)
I20260921 18:44:23.736397 4079 raft_consensus.cc:2454] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8fd3f01f1d425692d1f4a406422256 in term 2.
I20260921 18:44:23.736701 4274 leader_election.cc:302] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1af3525fb39a457a926165e1bb12608c, fc8fd3f01f1d425692d1f4a406422256; no voters: 06248013c41c42e99a2e41ab3dfa336c
I20260921 18:44:23.736951 4412 raft_consensus.cc:2781] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 2 FOLLOWER]: Leader election won for term 2
I20260921 18:44:23.737141 4412 raft_consensus.cc:686] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [term 2 LEADER]: Becoming Leader. State: Replica: fc8fd3f01f1d425692d1f4a406422256, State: Running, Role: LEADER
I20260921 18:44:23.737439 4412 consensus_queue.cc:237] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.739579 3932 catalog_manager.cc:5495] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 reported cstate change: term changed from 1 to 2, leader changed from 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130) to fc8fd3f01f1d425692d1f4a406422256 (127.3.36.131). New cstate: current_term: 2 leader_uuid: "fc8fd3f01f1d425692d1f4a406422256" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } health_report { overall_health: UNKNOWN } } }
I20260921 18:44:23.744879 4079 raft_consensus.cc:1261] T 419f624f4da64d41860b748b926a5713 P 1af3525fb39a457a926165e1bb12608c [term 2 FOLLOWER]: Refusing update from remote peer fc8fd3f01f1d425692d1f4a406422256: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260921 18:44:23.745313 4462 consensus_queue.cc:1035] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260921 18:44:23.745944 4274 consensus_peers.cc:487] T 419f624f4da64d41860b748b926a5713 P fc8fd3f01f1d425692d1f4a406422256 -> Peer 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307): Couldn't send request to peer 06248013c41c42e99a2e41ab3dfa336c. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:37307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260921 18:44:23.779203 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:37307
--local_ip_for_outbound_sockets=127.3.36.130
--tserver_master_addrs=127.3.36.190:37511
--webserver_port=37145
--webserver_interface=127.3.36.130
--builtin_ntp_servers=127.3.36.148:36637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260921 18:44:23.889436 4464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:23.889755 4464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:23.889874 4464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:23.894163 4464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:23.894285 4464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:44:23.898917 4464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:37307
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=37145
--tserver_master_addrs=127.3.36.190:37511
--never_fsync=true
--heap_profile_path=/tmp/kudu.4464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:23.899821 4464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:23.901134 4464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:23.910178 4470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:23.910209 4469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:23.910178 4472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:23.910461 4464 server_base.cc:1034] running on GCE node
I20260921 18:44:23.910837 4464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:23.911456 4464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:23.912600 4464 hybrid_clock.cc:648] HybridClock initialized: now 1790016263912573 us; error 40 us; skew 500 ppm
I20260921 18:44:23.914327 4464 webserver.cc:460] Webserver started at http://127.3.36.130:37145/ using document root <none> and password file <none>
I20260921 18:44:23.914837 4464 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:23.914933 4464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:23.918417 4464 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260921 18:44:23.920465 4478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:23.921620 4464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:44:23.921727 4464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "06248013c41c42e99a2e41ab3dfa336c"
format_stamp: "Formatted at 2026-09-21 18:44:22 on dist-test-slave-l4st"
I20260921 18:44:23.922240 4464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:23.937572 4464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:23.938202 4464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:23.938383 4464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:23.938907 4464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:23.940017 4485 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260921 18:44:23.943527 4464 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260921 18:44:23.943595 4464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260921 18:44:23.943681 4464 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260921 18:44:23.945251 4464 ts_tablet_manager.cc:610] Registered 1 tablets
I20260921 18:44:23.945304 4464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260921 18:44:23.945569 4485 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Bootstrap starting.
I20260921 18:44:23.955485 4464 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:37307
I20260921 18:44:23.955502 4592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:37307 every 8 connection(s)
I20260921 18:44:23.956733 4464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:44:23.961009 4485 log.cc:824] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:23.965276 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 4464
I20260921 18:44:23.965803 4593 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37511
I20260921 18:44:23.966045 4593 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:23.966562 4593 heartbeater.cc:511] Master 127.3.36.190:37511 requested a full tablet report, sending...
I20260921 18:44:23.968557 3932 ts_manager.cc:194] Re-registered known tserver with Master: 06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307)
I20260921 18:44:23.969527 3932 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:36767
I20260921 18:44:23.983338 4485 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260921 18:44:23.983760 4485 tablet_bootstrap.cc:492] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Bootstrap complete.
I20260921 18:44:23.985143 4485 ts_tablet_manager.cc:1397] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Time spent bootstrapping tablet: real 0.040s user 0.036s sys 0.000s
I20260921 18:44:23.988210 4485 raft_consensus.cc:348] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.988726 4485 raft_consensus.cc:729] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06248013c41c42e99a2e41ab3dfa336c, State: Initialized, Role: FOLLOWER
I20260921 18:44:23.989188 4485 consensus_queue.cc:260] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06248013c41c42e99a2e41ab3dfa336c" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 37307 } } peers { permanent_uuid: "fc8fd3f01f1d425692d1f4a406422256" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34861 } } peers { permanent_uuid: "1af3525fb39a457a926165e1bb12608c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42261 } }
I20260921 18:44:23.989811 4593 heartbeater.cc:503] Master 127.3.36.190:37511 was elected leader, sending a full tablet report...
I20260921 18:44:23.990195 4485 ts_tablet_manager.cc:1428] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260921 18:44:24.019846 4321 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:44:24.019938 4059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:44:24.020870 4527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
561543b86f6840a2ba43e1c5ff8503d5 | 127.3.36.190:37511 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
06248013c41c42e99a2e41ab3dfa336c | 127.3.36.130:37307 | HEALTHY | <none> | 0 | 0
1af3525fb39a457a926165e1bb12608c | 127.3.36.129:42261 | HEALTHY | <none> | 0 | 0
fc8fd3f01f1d425692d1f4a406422256 | 127.3.36.131:34861 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:42261
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:37307
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:34861
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:42261
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:37307
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:34861
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 419f624f4da64d41860b748b926a5713 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307): RUNNING
fc8fd3f01f1d425692d1f4a406422256 (127.3.36.131:34861): RUNNING [LEADER]
1af3525fb39a457a926165e1bb12608c (127.3.36.129:42261): RUNNING
All reported replicas are:
A = 06248013c41c42e99a2e41ab3dfa336c
B = fc8fd3f01f1d425692d1f4a406422256
C = 1af3525fb39a457a926165e1bb12608c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260921 18:44:24.046039 3218 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260921 18:44:24.241020 4527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:44:24.242442 4321 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:44:24.242472 4059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
561543b86f6840a2ba43e1c5ff8503d5 | 127.3.36.190:37511 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
06248013c41c42e99a2e41ab3dfa336c | 127.3.36.130:37307 | HEALTHY | <none> | 0 | 0
1af3525fb39a457a926165e1bb12608c | 127.3.36.129:42261 | HEALTHY | <none> | 0 | 0
fc8fd3f01f1d425692d1f4a406422256 | 127.3.36.131:34861 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:42261
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:37307
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:34861
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:42261
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:37307
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:34861
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 419f624f4da64d41860b748b926a5713 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
06248013c41c42e99a2e41ab3dfa336c (127.3.36.130:37307): RUNNING
fc8fd3f01f1d425692d1f4a406422256 (127.3.36.131:34861): RUNNING [LEADER]
1af3525fb39a457a926165e1bb12608c (127.3.36.129:42261): RUNNING
All reported replicas are:
A = 06248013c41c42e99a2e41ab3dfa336c
B = fc8fd3f01f1d425692d1f4a406422256
C = 1af3525fb39a457a926165e1bb12608c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260921 18:44:24.264102 3218 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260921 18:44:24.347302 4547 raft_consensus.cc:3037] T 419f624f4da64d41860b748b926a5713 P 06248013c41c42e99a2e41ab3dfa336c [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:44:24.356369 4646 mvcc.cc:205] Tried to move back new op lower bound from 7331906616424341504 to 7331906616271204352. Current Snapshot: MvccSnapshot[applied={T|T < 7331906616335040512}]
I20260921 18:44:24.535710 4527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:44:24.536538 4321 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:44:24.537376 4059 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
561543b86f6840a2ba43e1c5ff8503d5 | 127.3.36.190:37511 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
06248013c41c42e99a2e41ab3dfa336c | 127.3.36.130:37307 | HEALTHY | <none> | 0 | 0
1af3525fb39a457a926165e1bb12608c | 127.3.36.129:42261 | HEALTHY | <none> | 0 | 0
fc8fd3f01f1d425692d1f4a406422256 | 127.3.36.131:34861 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:42261
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:37307
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:34861
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:42261
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:37307
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:34861
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:36637 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260921 18:44:24.559235 3218 log_verifier.cc:126] Checking tablet 419f624f4da64d41860b748b926a5713
I20260921 18:44:24.580840 3218 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 419f624f4da64d41860b748b926a5713
I20260921 18:44:24.942667 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3997
I20260921 18:44:24.951627 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 4464
I20260921 18:44:24.957451 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 4259
I20260921 18:44:24.963002 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 3906
2026-09-21T18:44:24Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (2710 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20260921 18:44:24.986415 3218 test_util.cc:268] Using random seed: -1549934806
I20260921 18:44:24.987130 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:44:24.987183 3218 ts_itest-base.cc:113] --------------
I20260921 18:44:24.987205 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:44:24.987231 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:44:24.987254 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:44:24Z Disabled control of system clock
I20260921 18:44:24.995168 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:42281
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:42281 with env {}
W20260921 18:44:25.099906 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:25.100185 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:25.100260 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:25.104839 4691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:44:25.104913 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:25.104948 4691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:44:25.105000 4691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:44:25.109985 4691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:42281
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:25.111073 4691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:25.112478 4691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:25.120939 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:25.120934 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.121189 4691 server_base.cc:1034] running on GCE node
W20260921 18:44:25.121418 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.121929 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:25.122954 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:25.124123 4691 hybrid_clock.cc:648] HybridClock initialized: now 1790016265124105 us; error 32 us; skew 500 ppm
I20260921 18:44:25.125823 4691 webserver.cc:460] Webserver started at http://127.3.36.190:32923/ using document root <none> and password file <none>
I20260921 18:44:25.126330 4691 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:25.126411 4691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:25.126650 4691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:25.128397 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3d3d4837f42341be88af4e62b6864371"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.128933 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3d3d4837f42341be88af4e62b6864371"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.132266 4691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260921 18:44:25.134409 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.135502 4691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:44:25.135627 4691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3d3d4837f42341be88af4e62b6864371"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.135720 4691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:25.144169 4691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:25.144716 4691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:25.144881 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:25.151149 4691 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:42281
I20260921 18:44:25.151160 4757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:42281 every 8 connection(s)
I20260921 18:44:25.151926 4691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:44:25.154829 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:25.159188 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371: Bootstrap starting.
I20260921 18:44:25.159739 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 4691
I20260921 18:44:25.159852 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:44:25.161803 4758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:25.162621 4758 log.cc:824] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:25.164638 4758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371: No bootstrap required, opened a new log
I20260921 18:44:25.168512 4758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3d4837f42341be88af4e62b6864371" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42281 } }
I20260921 18:44:25.168756 4758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:25.168814 4758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3d4837f42341be88af4e62b6864371, State: Initialized, Role: FOLLOWER
I20260921 18:44:25.169215 4758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3d4837f42341be88af4e62b6864371" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42281 } }
I20260921 18:44:25.169356 4758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:44:25.169430 4758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:44:25.169559 4758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:25.170462 4758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3d4837f42341be88af4e62b6864371" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42281 } }
I20260921 18:44:25.170796 4758 leader_election.cc:302] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d3d4837f42341be88af4e62b6864371; no voters:
I20260921 18:44:25.171078 4758 leader_election.cc:288] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:44:25.171155 4763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:44:25.171377 4763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [term 1 LEADER]: Becoming Leader. State: Replica: 3d3d4837f42341be88af4e62b6864371, State: Running, Role: LEADER
I20260921 18:44:25.171697 4763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3d4837f42341be88af4e62b6864371" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42281 } }
I20260921 18:44:25.172137 4758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:44:25.173151 4764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d3d4837f42341be88af4e62b6864371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3d4837f42341be88af4e62b6864371" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42281 } } }
I20260921 18:44:25.173266 4764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [sys.catalog]: This master's current role is: LEADER
I20260921 18:44:25.173660 4772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:44:25.173622 4765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d3d4837f42341be88af4e62b6864371. Latest consensus state: current_term: 1 leader_uuid: "3d3d4837f42341be88af4e62b6864371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3d4837f42341be88af4e62b6864371" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42281 } } }
I20260921 18:44:25.173848 4765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371 [sys.catalog]: This master's current role is: LEADER
I20260921 18:44:25.176383 4772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:44:25.180896 4772 catalog_manager.cc:1347] Generated new cluster ID: c1109a5a2aa9407abd7cbc883184c5e4
I20260921 18:44:25.180962 4772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:44:25.189711 4772 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:44:25.190553 4772 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:44:25.198096 4772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d3d4837f42341be88af4e62b6864371: Generated new TSK 0
I20260921 18:44:25.198630 4772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:44:25.206105 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42281
--builtin_ntp_servers=127.3.36.148:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260921 18:44:25.311756 4782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:25.311985 4782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:25.312054 4782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:25.315919 4782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:25.315999 4782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:44:25.320498 4782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:42281
--never_fsync=true
--heap_profile_path=/tmp/kudu.4782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:25.321455 4782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:25.322715 4782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:25.331348 4788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.331548 4782 server_base.cc:1034] running on GCE node
W20260921 18:44:25.331341 4790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:25.331408 4787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.332064 4782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:25.332628 4782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:25.333801 4782 hybrid_clock.cc:648] HybridClock initialized: now 1790016265333781 us; error 27 us; skew 500 ppm
I20260921 18:44:25.335439 4782 webserver.cc:460] Webserver started at http://127.3.36.129:39861/ using document root <none> and password file <none>
I20260921 18:44:25.336000 4782 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:25.336081 4782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:25.336293 4782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:25.338121 4782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2b59277406044fc984fffda4ef77a363"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.338661 4782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2b59277406044fc984fffda4ef77a363"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.342166 4782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260921 18:44:25.344417 4796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.345572 4782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:44:25.345682 4782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2b59277406044fc984fffda4ef77a363"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.345777 4782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:25.357108 4782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:25.357714 4782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:25.357880 4782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:25.358374 4782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:25.359201 4782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:25.359265 4782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.359326 4782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:25.359377 4782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.367038 4782 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:36309
I20260921 18:44:25.367060 4909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:36309 every 8 connection(s)
I20260921 18:44:25.367852 4782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:44:25.371892 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 4782
I20260921 18:44:25.372027 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:44:25.374238 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42281
--builtin_ntp_servers=127.3.36.148:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260921 18:44:25.378366 4910 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42281
I20260921 18:44:25.378592 4910 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:25.379045 4910 heartbeater.cc:511] Master 127.3.36.190:42281 requested a full tablet report, sending...
I20260921 18:44:25.380625 4722 ts_manager.cc:194] Registered new tserver with Master: 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:25.382395 4722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:41833
W20260921 18:44:25.483104 4913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:25.483327 4913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:25.483363 4913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:25.487214 4913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:25.487339 4913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:44:25.491833 4913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:42281
--never_fsync=true
--heap_profile_path=/tmp/kudu.4913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:25.492654 4913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:25.493853 4913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:25.502686 4919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:25.502671 4918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.502880 4913 server_base.cc:1034] running on GCE node
W20260921 18:44:25.503018 4921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.503418 4913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:25.504123 4913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:25.505294 4913 hybrid_clock.cc:648] HybridClock initialized: now 1790016265505256 us; error 51 us; skew 500 ppm
I20260921 18:44:25.506973 4913 webserver.cc:460] Webserver started at http://127.3.36.130:44313/ using document root <none> and password file <none>
I20260921 18:44:25.507516 4913 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:25.507598 4913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:25.507794 4913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:25.509696 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9341d0fb8bc44c10ab9bad3c1818d831"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.510207 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9341d0fb8bc44c10ab9bad3c1818d831"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.513613 4913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:44:25.515810 4927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.516934 4913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:25.517068 4913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9341d0fb8bc44c10ab9bad3c1818d831"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.517166 4913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:25.536276 4913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:25.536895 4913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:25.537076 4913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:25.537667 4913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:25.538549 4913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:25.538614 4913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.538681 4913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:25.538721 4913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.546954 4913 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:38349
I20260921 18:44:25.546988 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:38349 every 8 connection(s)
I20260921 18:44:25.547832 4913 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:44:25.549883 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 4913
I20260921 18:44:25.549999 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:44:25.555341 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42281
--builtin_ntp_servers=127.3.36.148:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260921 18:44:25.560932 5041 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42281
I20260921 18:44:25.561188 5041 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:25.561818 5041 heartbeater.cc:511] Master 127.3.36.190:42281 requested a full tablet report, sending...
I20260921 18:44:25.562844 4722 ts_manager.cc:194] Registered new tserver with Master: 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349)
I20260921 18:44:25.563380 4722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:59871
W20260921 18:44:25.666232 5044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:25.666491 5044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:25.666532 5044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:25.670785 5044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:25.670907 5044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:44:25.675637 5044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:42281
--never_fsync=true
--heap_profile_path=/tmp/kudu.5044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:25.676573 5044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:25.677932 5044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:25.686811 5049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.686909 5044 server_base.cc:1034] running on GCE node
W20260921 18:44:25.686818 5050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:25.686847 5052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:25.687441 5044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:25.688087 5044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:25.689250 5044 hybrid_clock.cc:648] HybridClock initialized: now 1790016265689235 us; error 29 us; skew 500 ppm
I20260921 18:44:25.691038 5044 webserver.cc:460] Webserver started at http://127.3.36.131:34287/ using document root <none> and password file <none>
I20260921 18:44:25.691578 5044 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:25.691637 5044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:25.691864 5044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:25.693758 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "beb9da40af4040bfba9e185697c750ca"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.694272 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "beb9da40af4040bfba9e185697c750ca"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.697808 5044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:44:25.700050 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.701184 5044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:25.701270 5044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "beb9da40af4040bfba9e185697c750ca"
format_stamp: "Formatted at 2026-09-21 18:44:25 on dist-test-slave-l4st"
I20260921 18:44:25.701345 5044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:25.718710 5044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:25.719393 5044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:25.719599 5044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:25.720145 5044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:25.721071 5044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:25.721131 5044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.721194 5044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:25.721240 5044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:25.729297 5044 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:34241
I20260921 18:44:25.729326 5171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:34241 every 8 connection(s)
I20260921 18:44:25.730170 5044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:44:25.731061 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 5044
I20260921 18:44:25.731171 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:44:25.739537 5172 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42281
I20260921 18:44:25.739744 5172 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:25.740190 5172 heartbeater.cc:511] Master 127.3.36.190:42281 requested a full tablet report, sending...
I20260921 18:44:25.741106 4722 ts_manager.cc:194] Registered new tserver with Master: beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:25.741648 4722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:42861
I20260921 18:44:25.744863 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:44:25.746304 3218 test_util.cc:268] Using random seed: -1549174917
I20260921 18:44:25.755784 4722 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41796:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260921 18:44:25.756783 4722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:44:25.773393 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 596bd65c0a4d44f5a1cc6390db4f0757 (DEFAULT_TABLE table=test-workload [id=22774889ece84dac91bcf3707a319d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:25.773308 4844 tablet_service.cc:1467] Processing CreateTablet for tablet 596bd65c0a4d44f5a1cc6390db4f0757 (DEFAULT_TABLE table=test-workload [id=22774889ece84dac91bcf3707a319d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:25.774328 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596bd65c0a4d44f5a1cc6390db4f0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:25.774034 4975 tablet_service.cc:1467] Processing CreateTablet for tablet 596bd65c0a4d44f5a1cc6390db4f0757 (DEFAULT_TABLE table=test-workload [id=22774889ece84dac91bcf3707a319d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:25.774375 4844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596bd65c0a4d44f5a1cc6390db4f0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:25.774942 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596bd65c0a4d44f5a1cc6390db4f0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:25.780395 5189 tablet_bootstrap.cc:492] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca: Bootstrap starting.
I20260921 18:44:25.781081 5191 tablet_bootstrap.cc:492] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: Bootstrap starting.
I20260921 18:44:25.781759 5190 tablet_bootstrap.cc:492] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363: Bootstrap starting.
I20260921 18:44:25.782686 5189 tablet_bootstrap.cc:654] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:25.783121 5191 tablet_bootstrap.cc:654] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:25.783401 5190 tablet_bootstrap.cc:654] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:25.783490 5189 log.cc:824] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:25.783910 5191 log.cc:824] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:25.784238 5190 log.cc:824] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:25.785879 5191 tablet_bootstrap.cc:492] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: No bootstrap required, opened a new log
I20260921 18:44:25.786111 5191 ts_tablet_manager.cc:1397] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:44:25.786239 5190 tablet_bootstrap.cc:492] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363: No bootstrap required, opened a new log
I20260921 18:44:25.786455 5190 ts_tablet_manager.cc:1397] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:44:25.786706 5189 tablet_bootstrap.cc:492] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca: No bootstrap required, opened a new log
I20260921 18:44:25.786863 5189 ts_tablet_manager.cc:1397] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260921 18:44:25.789913 5190 raft_consensus.cc:348] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.790144 5190 raft_consensus.cc:374] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:25.790215 5190 raft_consensus.cc:729] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b59277406044fc984fffda4ef77a363, State: Initialized, Role: FOLLOWER
I20260921 18:44:25.790652 5190 consensus_queue.cc:260] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.790755 5191 raft_consensus.cc:348] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.791052 5191 raft_consensus.cc:374] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:25.791148 5191 raft_consensus.cc:729] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9341d0fb8bc44c10ab9bad3c1818d831, State: Initialized, Role: FOLLOWER
I20260921 18:44:25.791280 4910 heartbeater.cc:503] Master 127.3.36.190:42281 was elected leader, sending a full tablet report...
I20260921 18:44:25.791285 5189 raft_consensus.cc:348] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.791535 5189 raft_consensus.cc:374] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:25.791580 5189 raft_consensus.cc:729] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb9da40af4040bfba9e185697c750ca, State: Initialized, Role: FOLLOWER
I20260921 18:44:25.791671 5190 ts_tablet_manager.cc:1428] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:44:25.791625 5191 consensus_queue.cc:260] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.792083 5195 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:25.792232 5195 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.792701 5191 ts_tablet_manager.cc:1428] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:44:25.792946 5198 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:25.792997 5041 heartbeater.cc:503] Master 127.3.36.190:42281 was elected leader, sending a full tablet report...
I20260921 18:44:25.793109 5198 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.793411 5195 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:25.794502 5189 consensus_queue.cc:260] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.795578 5189 ts_tablet_manager.cc:1428] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20260921 18:44:25.795909 5198 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:25.796109 5172 heartbeater.cc:503] Master 127.3.36.190:42281 was elected leader, sending a full tablet report...
W20260921 18:44:25.798681 5042 tablet.cc:2369] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:44:25.799158 5203 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:25.799293 5203 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.800329 5203 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
W20260921 18:44:25.801129 4800 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.3.36.131:34241 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260921 18:44:25.801222 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:25.801400 4864 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in term 0.
W20260921 18:44:25.801802 4931 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.3.36.131:34241 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20260921 18:44:25.802145 4931 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831; no voters:
I20260921 18:44:25.802574 5196 raft_consensus.cc:2781] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:44:25.802667 5196 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:44:25.802758 5196 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:25.802877 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:25.803063 5126 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in term 0.
I20260921 18:44:25.804095 5196 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.804162 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:25.804633 4995 raft_consensus.cc:2379] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b59277406044fc984fffda4ef77a363 in current term 1: Already voted for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in this term.
I20260921 18:44:25.804752 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca"
I20260921 18:44:25.804862 5126 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:25.806052 5126 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in term 1.
I20260921 18:44:25.806380 4931 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca; no voters:
I20260921 18:44:25.806726 5198 raft_consensus.cc:2781] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:44:25.806922 5198 raft_consensus.cc:686] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 LEADER]: Becoming Leader. State: Replica: 9341d0fb8bc44c10ab9bad3c1818d831, State: Running, Role: LEADER
I20260921 18:44:25.806921 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:25.807224 5198 consensus_queue.cc:237] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:25.808923 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:25.809091 5126 raft_consensus.cc:2379] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b59277406044fc984fffda4ef77a363 in current term 1: Already voted for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in this term.
I20260921 18:44:25.809500 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:25.809710 4864 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 0 FOLLOWER]: Advancing to term 1
W20260921 18:44:25.809988 5062 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.3.36.129:36309 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20260921 18:44:25.809729 4721 catalog_manager.cc:5495] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130). New cstate: current_term: 1 leader_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } health_report { overall_health: UNKNOWN } } }
I20260921 18:44:25.810401 5208 consensus_queue.cc:1035] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:44:25.811715 5197 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260921 18:44:25.812602 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:25.812718 4864 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beb9da40af4040bfba9e185697c750ca in term 1.
I20260921 18:44:25.813066 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b59277406044fc984fffda4ef77a363, beb9da40af4040bfba9e185697c750ca; no voters: 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:25.813187 5210 consensus_queue.cc:1035] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:44:25.813517 5200 raft_consensus.cc:2741] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260921 18:44:25.813895 4863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363"
I20260921 18:44:25.813647 5196 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 1 election: Requested vote from peers beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
W20260921 18:44:25.839021 5214 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:44:25.839103 5214 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260921 18:44:26.156523 5200 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:26.156665 5200 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:26.157157 5200 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:26.157384 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:26.157541 4863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:26.158089 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:26.158308 5200 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:26.677618 4863 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:44:26.677618 5126 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:44:26.677702 5126 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260921 18:44:26.677702 4863 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260921 18:44:26.888372 5223 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:26.888506 5223 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:26.889575 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:26.889628 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:26.890062 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:26.890348 5223 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:26.892180 5226 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=1, attempt_no=0}) took 1057 ms. Trace:
I20260921 18:44:26.892300 5226 rpcz_store.cc:276] 0921 18:44:25.834870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:25.834926 (+ 56us) service_pool.cc:227] Handling call
0921 18:44:26.892166 (+1057240us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:25.837267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:25.837357 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:25.837370 (+ 13us) write_op.cc:435] Acquired schema lock
0921 18:44:25.837373 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:25.837968 (+ 595us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:25.838045 (+ 77us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:25.838053 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:25.838841 (+ 788us) tablet.cc:699] Row locks acquired
0921 18:44:25.838844 (+ 3us) write_op.cc:260] PREPARE: finished
0921 18:44:25.838955 (+ 111us) write_op.cc:270] Start()
0921 18:44:26.675448 (+836493us) write_op.cc:276] Timestamp: P: 1790016265838884 usec, L: 0
0921 18:44:26.675453 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:44:26.675906 (+ 453us) log.cc:842] Serialized 345 byte log entry
0921 18:44:26.889561 (+213655us) op_driver.cc:444] REPLICATION: finished
0921 18:44:26.890127 (+ 566us) write_op.cc:301] APPLY: starting
0921 18:44:26.890182 (+ 55us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:26.890400 (+ 218us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:26.890416 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:26.890719 (+ 303us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:26.890874 (+ 155us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:26.890894 (+ 20us) write_op.cc:312] APPLY: finished
0921 18:44:26.891134 (+ 240us) log.cc:842] Serialized 22 byte log entry
0921 18:44:26.891860 (+ 726us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:26.891899 (+ 39us) write_op.cc:454] Released schema lock
0921 18:44:26.891907 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1494528,"child_traces":[["op",{"apply.queue_time_us":493,"mutex_wait_us":80,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":2735,"prepare.run_wall_time_us":838878,"raft.queue_time_us":120,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":214062,"spinlock_wait_cycles":97536,"thread_start_us":58,"threads_started":1}]]}
W20260921 18:44:26.896031 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.918s user 0.001s sys 0.000s
W20260921 18:44:26.896027 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.918s user 0.001s sys 0.000s
I20260921 18:44:27.447486 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:27.447659 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:27.448335 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:27.448690 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:27.448791 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:27.449267 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:27.612255 5227 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:27.613791 5236 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=0, attempt_no=0}) took 1779 ms. Trace:
I20260921 18:44:27.613891 5236 rpcz_store.cc:276] 0921 18:44:25.834728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:25.834784 (+ 56us) service_pool.cc:227] Handling call
0921 18:44:27.613776 (+1778992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:26.676162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:26.676291 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:26.676296 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:26.676297 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:26.676403 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:26.676432 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:26.676435 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:26.676478 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:44:26.676479 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:26.676515 (+ 36us) write_op.cc:270] Start()
0921 18:44:26.743666 (+ 67151us) write_op.cc:276] Timestamp: P: 1790016266676483 usec, L: 0
0921 18:44:26.743669 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:26.743847 (+ 178us) log.cc:842] Serialized 345 byte log entry
0921 18:44:27.613142 (+869295us) op_driver.cc:444] REPLICATION: finished
0921 18:44:27.613403 (+ 261us) write_op.cc:301] APPLY: starting
0921 18:44:27.613417 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:27.613445 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:27.613458 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:27.613530 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:27.613560 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:27.613564 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:27.613626 (+ 62us) log.cc:842] Serialized 22 byte log entry
0921 18:44:27.613695 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:27.613705 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:27.613710 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"num_ops":1,"prepare.queue_time_us":836916,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":67739,"replication_time_us":869455,"thread_start_us":90,"threads_started":1}]]}
I20260921 18:44:27.930567 5227 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:27.930686 5227 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:27.931187 5227 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:27.931643 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:27.931675 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:27.932101 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:27.932268 5227 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:27.933197 5236 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=2, attempt_no=0}) took 1038 ms. Trace:
I20260921 18:44:27.933288 5236 rpcz_store.cc:276] 0921 18:44:26.895115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:26.895179 (+ 64us) service_pool.cc:227] Handling call
0921 18:44:27.933188 (+1038009us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:26.895569 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:26.895640 (+ 71us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:26.895643 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:26.895644 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:26.895692 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:26.895697 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:26.895699 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:26.895726 (+ 27us) tablet.cc:699] Row locks acquired
0921 18:44:26.895727 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:26.895747 (+ 20us) write_op.cc:270] Start()
0921 18:44:27.488868 (+593121us) write_op.cc:276] Timestamp: P: 1790016266895698 usec, L: 0
0921 18:44:27.488871 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:27.489129 (+ 258us) log.cc:842] Serialized 354 byte log entry
0921 18:44:27.931794 (+442665us) op_driver.cc:444] REPLICATION: finished
0921 18:44:27.932259 (+ 465us) write_op.cc:301] APPLY: starting
0921 18:44:27.932274 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:27.932301 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:27.932311 (+ 10us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:27.932357 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:27.932385 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:27.932388 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:44:27.932486 (+ 98us) log.cc:842] Serialized 22 byte log entry
0921 18:44:27.933104 (+ 618us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:27.933126 (+ 22us) write_op.cc:454] Released schema lock
0921 18:44:27.933131 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":411,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":542,"prepare.run_wall_time_us":593606,"replication_time_us":442904}]]}
W20260921 18:44:27.934213 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.887s user 0.001s sys 0.000s
W20260921 18:44:27.934213 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.887s user 0.001s sys 0.000s
I20260921 18:44:28.170497 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:28.854357 5227 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:28.854521 5227 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:28.854969 5227 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:28.855315 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:28.855341 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:28.855751 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:28.855957 5227 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:28.857261 5243 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=3, attempt_no=0}) took 1242 ms. Trace:
W20260921 18:44:28.857434 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.773s user 0.001s sys 0.000s
I20260921 18:44:28.857357 5243 rpcz_store.cc:276] 0921 18:44:27.615050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:27.615109 (+ 59us) service_pool.cc:227] Handling call
0921 18:44:28.857249 (+1242140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:27.615532 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:27.615614 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:27.615617 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:27.615618 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:27.615675 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:27.615679 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:27.615681 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:27.615706 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:44:27.615707 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:27.615730 (+ 23us) write_op.cc:270] Start()
0921 18:44:28.252856 (+637126us) write_op.cc:276] Timestamp: P: 1790016267615724 usec, L: 0
0921 18:44:28.252859 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:28.253087 (+ 228us) log.cc:842] Serialized 354 byte log entry
0921 18:44:28.855730 (+602643us) op_driver.cc:444] REPLICATION: finished
0921 18:44:28.856055 (+ 325us) write_op.cc:301] APPLY: starting
0921 18:44:28.856071 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:28.856102 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:28.856113 (+ 11us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:28.856169 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:28.856192 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:28.856195 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:44:28.856241 (+ 46us) log.cc:842] Serialized 22 byte log entry
0921 18:44:28.857178 (+ 937us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:28.857189 (+ 11us) write_op.cc:454] Released schema lock
0921 18:44:28.857193 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":249,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":637628,"replication_time_us":602847,"spinlock_wait_cycles":3328,"thread_start_us":67,"threads_started":1}]]}
W20260921 18:44:28.858232 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.773s user 0.001s sys 0.000s
I20260921 18:44:29.009868 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:29.010035 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:29.010524 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:29.011013 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:29.011220 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:29.011632 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:29.039857 5227 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:29.039983 5227 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:29.040436 5227 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:29.040758 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:29.040789 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:29.041235 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:29.041407 5227 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:29.041675 5243 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=4, attempt_no=0}) took 1106 ms. Trace:
I20260921 18:44:29.041759 5243 rpcz_store.cc:276] 0921 18:44:27.934734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:27.934787 (+ 53us) service_pool.cc:227] Handling call
0921 18:44:29.041668 (+1106881us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:28.253181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:28.253280 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:28.253286 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:28.253287 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:28.253359 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:28.253370 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:28.253373 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:28.253418 (+ 45us) tablet.cc:699] Row locks acquired
0921 18:44:28.253419 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:28.253446 (+ 27us) write_op.cc:270] Start()
0921 18:44:28.501585 (+248139us) write_op.cc:276] Timestamp: P: 1790016268253403 usec, L: 0
0921 18:44:28.501589 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:28.501769 (+ 180us) log.cc:842] Serialized 354 byte log entry
0921 18:44:29.041078 (+539309us) op_driver.cc:444] REPLICATION: finished
0921 18:44:29.041208 (+ 130us) write_op.cc:301] APPLY: starting
0921 18:44:29.041220 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:29.041247 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:29.041262 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:29.041312 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:29.041340 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:29.041346 (+ 6us) write_op.cc:312] APPLY: finished
0921 18:44:29.041400 (+ 54us) log.cc:842] Serialized 22 byte log entry
0921 18:44:29.041609 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:29.041618 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:29.041622 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":92,"num_ops":1,"prepare.queue_time_us":318098,"prepare.run_cpu_time_us":596,"prepare.run_wall_time_us":248662,"replication_time_us":539472}]]}
I20260921 18:44:30.364207 5248 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:30.364369 5248 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:30.364898 5248 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:30.365334 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:30.365615 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:30.365995 5253 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=5, attempt_no=0}) took 1507 ms. Trace:
I20260921 18:44:30.366040 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:30.366086 5253 rpcz_store.cc:276] 0921 18:44:28.858632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:28.858685 (+ 53us) service_pool.cc:227] Handling call
0921 18:44:30.365979 (+1507294us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:28.859019 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:28.859087 (+ 68us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:28.859090 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:28.859091 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:28.859145 (+ 54us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:28.859150 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:28.859152 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:28.859175 (+ 23us) tablet.cc:699] Row locks acquired
0921 18:44:28.859176 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:28.859198 (+ 22us) write_op.cc:270] Start()
0921 18:44:29.809328 (+950130us) write_op.cc:276] Timestamp: P: 1790016268859152 usec, L: 0
0921 18:44:29.809331 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:29.809541 (+ 210us) log.cc:842] Serialized 354 byte log entry
0921 18:44:30.365347 (+555806us) op_driver.cc:444] REPLICATION: finished
0921 18:44:30.365623 (+ 276us) write_op.cc:301] APPLY: starting
0921 18:44:30.365642 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:30.365672 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:30.365688 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:30.365742 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:30.365772 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:30.365776 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:30.365831 (+ 55us) log.cc:842] Serialized 22 byte log entry
0921 18:44:30.365901 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:30.365910 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:30.365915 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":550,"prepare.run_wall_time_us":950605,"replication_time_us":555996,"thread_start_us":82,"threads_started":1}]]}
I20260921 18:44:30.366850 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 13) took 1357 ms. Trace:
I20260921 18:44:30.366915 4955 rpcz_store.cc:276] 0921 18:44:29.009088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:29.009169 (+ 81us) service_pool.cc:227] Handling call
0921 18:44:29.009297 (+ 128us) tablet_service.cc:2780] Created scanner 92a152d50b0e47b8a311e35b3191c710 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 3226223d652a423da38b15d5c5d08e16
0921 18:44:29.009540 (+ 243us) tablet_service.cc:2920] Creating iterator
0921 18:44:29.009561 (+ 21us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:29.809486 (+799925us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:30.366013 (+556527us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1356440 microseconds
0921 18:44:30.366099 (+ 86us) tablet_service.cc:2945] Iterator created
0921 18:44:30.366383 (+ 284us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:30.366408 (+ 25us) tablet_service.cc:3010] has_more: true
0921 18:44:30.366438 (+ 30us) tablet_service.cc:3027] Continuing scan request
0921 18:44:30.366483 (+ 45us) tablet_service.cc:3091] Found scanner 92a152d50b0e47b8a311e35b3191c710 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 3226223d652a423da38b15d5c5d08e16
0921 18:44:30.366838 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20260921 18:44:30.367319 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 12) took 1358 ms. Trace:
I20260921 18:44:30.367401 4953 rpcz_store.cc:276] 0921 18:44:29.008741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:29.008826 (+ 85us) service_pool.cc:227] Handling call
0921 18:44:29.009021 (+ 195us) tablet_service.cc:2780] Created scanner 280612ea80d74128aab63d0be155c2f4 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is d6d20aa309aa40d4ae05dd9294a53787
0921 18:44:29.009217 (+ 196us) tablet_service.cc:2920] Creating iterator
0921 18:44:29.009240 (+ 23us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:29.809513 (+800273us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:30.366634 (+557121us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1357382 microseconds
0921 18:44:30.366700 (+ 66us) tablet_service.cc:2945] Iterator created
0921 18:44:30.366933 (+ 233us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:30.366954 (+ 21us) tablet_service.cc:3010] has_more: true
0921 18:44:30.366978 (+ 24us) tablet_service.cc:3027] Continuing scan request
0921 18:44:30.367021 (+ 43us) tablet_service.cc:3091] Found scanner 280612ea80d74128aab63d0be155c2f4 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is d6d20aa309aa40d4ae05dd9294a53787
0921 18:44:30.367312 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:44:30.367678 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.359s user 0.000s sys 0.000s
W20260921 18:44:30.367996 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.360s user 0.001s sys 0.000s
I20260921 18:44:30.677079 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:30.677361 4863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.130:51975 (request call id 756) took 1666 ms. Trace:
I20260921 18:44:30.677456 4863 rpcz_store.cc:276] 0921 18:44:29.011216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:29.011262 (+ 46us) service_pool.cc:227] Handling call
0921 18:44:29.011423 (+ 161us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:44:29.011525 (+ 102us) raft_consensus.cc:1520] Early marking committed up to index 0
0921 18:44:29.011531 (+ 6us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:44:29.011633 (+ 102us) write_op.cc:270] Start()
0921 18:44:29.808771 (+797138us) write_op.cc:276] Timestamp: P: 1790016267615724 usec, L: 0
0921 18:44:29.809139 (+ 368us) write_op.cc:270] Start()
0921 18:44:30.676283 (+867144us) write_op.cc:276] Timestamp: P: 1790016268253403 usec, L: 0
0921 18:44:30.676738 (+ 455us) log.cc:842] Serialized 708 byte log entry
0921 18:44:30.676874 (+ 136us) raft_consensus.cc:1631] Marking committed up to 5
0921 18:44:30.676995 (+ 121us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:44:30.677025 (+ 30us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:44:30.677297 (+ 272us) raft_consensus.cc:1671] finished
0921 18:44:30.677299 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:44:30.677345 (+ 46us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:29.809058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:29.809155 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:29.809161 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:29.809163 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:29.809237 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:29.809246 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:29.809248 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:29.809320 (+ 72us) tablet.cc:699] Row locks acquired
0921 18:44:29.809321 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:30.676904 (+867583us) op_driver.cc:444] REPLICATION: finished
0921 18:44:30.677084 (+ 180us) write_op.cc:301] APPLY: starting
0921 18:44:30.677099 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:30.677128 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:30.677129 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:30.677181 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:30.677211 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:30.677238 (+ 27us) write_op.cc:312] APPLY: finished
0921 18:44:30.677303 (+ 65us) log.cc:842] Serialized 22 byte log entry
0921 18:44:30.677370 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:30.677376 (+ 6us) write_op.cc:454] Released schema lock
0921 18:44:30.677380 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0921 18:44:30.677423 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:30.677424 (+ 1us) write_op.cc:454] Released schema lock
Related trace 'op':
0921 18:44:30.676634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:30.676732 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:30.676736 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:30.676737 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:30.676800 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:30.676828 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:30.676830 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:30.676863 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:44:30.676864 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":137,"child_traces":[["op",{"apply.queue_time_us":139,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":372,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":492,"prepare.run_wall_time_us":491,"replication_time_us":1665259,"thread_start_us":143,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":24,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":299,"prepare.run_wall_time_us":299,"thread_start_us":74,"threads_started":1}]]}
I20260921 18:44:30.944845 5248 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:30.946568 5261 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=6, attempt_no=0}) took 1903 ms. Trace:
I20260921 18:44:30.946655 5261 rpcz_store.cc:276] 0921 18:44:29.042871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:29.042927 (+ 56us) service_pool.cc:227] Handling call
0921 18:44:30.946549 (+1903622us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:29.809627 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:29.809714 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:29.809720 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:29.809723 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:29.809785 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:29.809791 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:29.809794 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:29.809826 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:44:29.809827 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:29.809855 (+ 28us) write_op.cc:270] Start()
0921 18:44:29.859028 (+ 49173us) write_op.cc:276] Timestamp: P: 1790016269809813 usec, L: 0
0921 18:44:29.859032 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:29.859182 (+ 150us) log.cc:842] Serialized 354 byte log entry
0921 18:44:30.945794 (+1086612us) op_driver.cc:444] REPLICATION: finished
0921 18:44:30.946038 (+ 244us) write_op.cc:301] APPLY: starting
0921 18:44:30.946057 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:30.946086 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:30.946101 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:30.946161 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:30.946189 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:30.946193 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:30.946277 (+ 84us) log.cc:842] Serialized 22 byte log entry
0921 18:44:30.946464 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:30.946474 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:30.946479 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":190,"num_ops":1,"prepare.queue_time_us":766448,"prepare.run_cpu_time_us":529,"prepare.run_wall_time_us":49608,"replication_time_us":1086742,"thread_start_us":86,"threads_started":1}]]}
I20260921 18:44:31.696262 5248 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:31.696396 5248 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:31.696957 5248 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:31.697288 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:31.697405 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:31.697887 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:31.698879 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 17) took 1179 ms. Trace:
I20260921 18:44:31.698949 4955 rpcz_store.cc:276] 0921 18:44:30.519012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:30.519070 (+ 58us) service_pool.cc:227] Handling call
0921 18:44:30.519227 (+ 157us) tablet_service.cc:2780] Created scanner 32b1c1169a07407da6955ae5a95a6db3 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 46c8ef6d3baa4968acb514ee194b6a0e
0921 18:44:30.519423 (+ 196us) tablet_service.cc:2920] Creating iterator
0921 18:44:30.519459 (+ 36us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:30.689622 (+170163us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:31.698045 (+1008423us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1178576 microseconds
0921 18:44:31.698118 (+ 73us) tablet_service.cc:2945] Iterator created
0921 18:44:31.698427 (+ 309us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:31.698452 (+ 25us) tablet_service.cc:3010] has_more: true
0921 18:44:31.698479 (+ 27us) tablet_service.cc:3027] Continuing scan request
0921 18:44:31.698535 (+ 56us) tablet_service.cc:3091] Found scanner 32b1c1169a07407da6955ae5a95a6db3 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 46c8ef6d3baa4968acb514ee194b6a0e
0921 18:44:31.698868 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":67072}
I20260921 18:44:31.699163 5267 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=7, attempt_no=0}) took 1331 ms. Trace:
I20260921 18:44:31.699240 5267 rpcz_store.cc:276] 0921 18:44:30.367617 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:30.367680 (+ 63us) service_pool.cc:227] Handling call
0921 18:44:31.699148 (+1331468us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:30.368129 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:30.368198 (+ 69us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:30.368203 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:30.368204 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:30.368282 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:30.368288 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:30.368290 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:30.368318 (+ 28us) tablet.cc:699] Row locks acquired
0921 18:44:30.368319 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:30.368350 (+ 31us) write_op.cc:270] Start()
0921 18:44:30.689492 (+321142us) write_op.cc:276] Timestamp: P: 1790016270368302 usec, L: 0
0921 18:44:30.689494 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:44:30.689668 (+ 174us) log.cc:842] Serialized 354 byte log entry
0921 18:44:31.697561 (+1007893us) op_driver.cc:444] REPLICATION: finished
0921 18:44:31.697762 (+ 201us) write_op.cc:301] APPLY: starting
0921 18:44:31.697779 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:31.697808 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:31.697823 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:31.697871 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:31.697897 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:31.697901 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:31.697955 (+ 54us) log.cc:842] Serialized 22 byte log entry
0921 18:44:31.699044 (+ 1089us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:31.699054 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:31.699059 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":158,"num_ops":1,"prepare.queue_time_us":160,"prepare.run_cpu_time_us":540,"prepare.run_wall_time_us":321609,"replication_time_us":1008047,"thread_start_us":127,"threads_started":2}]]}
W20260921 18:44:31.699510 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.181s user 0.001s sys 0.000s
I20260921 18:44:31.699826 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 16) took 1180 ms. Trace:
I20260921 18:44:31.699901 4953 rpcz_store.cc:276] 0921 18:44:30.518943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:30.519019 (+ 76us) service_pool.cc:227] Handling call
0921 18:44:30.519227 (+ 208us) tablet_service.cc:2780] Created scanner 56cf70969ffd424884be170d74577038 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is b5ee6437e4fa42b1b7d70e81dcd232a9
0921 18:44:30.519423 (+ 196us) tablet_service.cc:2920] Creating iterator
0921 18:44:30.519458 (+ 35us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:30.689622 (+170164us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:31.699080 (+1009458us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1179611 microseconds
0921 18:44:31.699137 (+ 57us) tablet_service.cc:2945] Iterator created
0921 18:44:31.699394 (+ 257us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:31.699418 (+ 24us) tablet_service.cc:3010] has_more: true
0921 18:44:31.699442 (+ 24us) tablet_service.cc:3027] Continuing scan request
0921 18:44:31.699488 (+ 46us) tablet_service.cc:3091] Found scanner 56cf70969ffd424884be170d74577038 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is b5ee6437e4fa42b1b7d70e81dcd232a9
0921 18:44:31.699817 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":33152}
W20260921 18:44:31.700309 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.182s user 0.000s sys 0.000s
I20260921 18:44:32.117827 5248 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:32.119474 5267 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=8, attempt_no=0}) took 1171 ms. Trace:
I20260921 18:44:32.119606 5267 rpcz_store.cc:276] 0921 18:44:30.947793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:30.947862 (+ 69us) service_pool.cc:227] Handling call
0921 18:44:32.119430 (+1171568us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:30.948210 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:30.948286 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:30.948289 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:30.948290 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:30.948337 (+ 47us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:30.948341 (+ 4us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:30.948343 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:30.948368 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:44:30.948369 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:30.948392 (+ 23us) write_op.cc:270] Start()
0921 18:44:31.337575 (+389183us) write_op.cc:276] Timestamp: P: 1790016270948334 usec, L: 0
0921 18:44:31.337579 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:31.337772 (+ 193us) log.cc:842] Serialized 354 byte log entry
0921 18:44:32.118606 (+780834us) op_driver.cc:444] REPLICATION: finished
0921 18:44:32.118687 (+ 81us) write_op.cc:301] APPLY: starting
0921 18:44:32.118725 (+ 38us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:32.118749 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:32.118763 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:32.118806 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:32.118848 (+ 42us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:32.118852 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:32.118916 (+ 64us) log.cc:842] Serialized 22 byte log entry
0921 18:44:32.118972 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:32.119346 (+ 374us) write_op.cc:454] Released schema lock
0921 18:44:32.119353 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":541,"prepare.run_wall_time_us":389642,"replication_time_us":781008}]]}
I20260921 18:44:32.473133 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:32.473289 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:32.473758 4863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.130:51975 (request call id 757) took 1794 ms. Trace:
I20260921 18:44:32.473873 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:32.473826 4863 rpcz_store.cc:276] 0921 18:44:30.679274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:30.679342 (+ 68us) service_pool.cc:227] Handling call
0921 18:44:30.679467 (+ 125us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:44:30.679532 (+ 65us) raft_consensus.cc:1520] Early marking committed up to index 6
0921 18:44:30.679585 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:44:30.679670 (+ 85us) write_op.cc:270] Start()
0921 18:44:31.667821 (+988151us) write_op.cc:276] Timestamp: P: 1790016268859152 usec, L: 0
0921 18:44:31.668122 (+ 301us) write_op.cc:270] Start()
0921 18:44:32.472274 (+804152us) write_op.cc:276] Timestamp: P: 1790016269809813 usec, L: 0
0921 18:44:32.472730 (+ 456us) log.cc:842] Serialized 708 byte log entry
0921 18:44:32.472863 (+ 133us) raft_consensus.cc:1631] Marking committed up to 7
0921 18:44:32.473000 (+ 137us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:44:32.473025 (+ 25us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:44:32.473700 (+ 675us) raft_consensus.cc:1671] finished
0921 18:44:32.473702 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:44:32.473735 (+ 33us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:31.668121 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:31.668207 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:31.668210 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:31.668212 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:31.668274 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:31.668283 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:31.668285 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:31.668317 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:44:31.668318 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:32.472892 (+804574us) op_driver.cc:444] REPLICATION: finished
Related trace 'op':
0921 18:44:32.472760 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:32.472845 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:32.472848 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:32.472850 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:32.472913 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:32.472921 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:32.472923 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:32.472979 (+ 56us) tablet.cc:699] Row locks acquired
0921 18:44:32.472981 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":139,"child_traces":[["op",{"apply.queue_time_us":1186,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":183,"prepare.run_cpu_time_us":263,"prepare.run_wall_time_us":264,"replication_time_us":1793208,"thread_start_us":141,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":362,"prepare.run_cpu_time_us":302,"prepare.run_wall_time_us":301,"thread_start_us":104,"threads_started":1}]]}
I20260921 18:44:32.474222 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:32.475032 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:32.475414 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:32.984921 5248 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:32.985065 5248 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:32.985612 5248 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:32.985936 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:32.986016 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:32.986413 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:32.986598 5274 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=9, attempt_no=0}) took 1286 ms. Trace:
I20260921 18:44:32.986645 5248 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:32.986680 5274 rpcz_store.cc:276] 0921 18:44:31.700507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:31.700570 (+ 63us) service_pool.cc:227] Handling call
0921 18:44:32.986584 (+1286014us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:31.700874 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:31.700956 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:31.700959 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:31.700960 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:31.701017 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:31.701023 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:31.701025 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:31.701049 (+ 24us) tablet.cc:699] Row locks acquired
0921 18:44:31.701049 (+ 0us) write_op.cc:260] PREPARE: finished
0921 18:44:31.701072 (+ 23us) write_op.cc:270] Start()
0921 18:44:32.541213 (+840141us) write_op.cc:276] Timestamp: P: 1790016271701036 usec, L: 0
0921 18:44:32.541217 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:32.541460 (+ 243us) log.cc:842] Serialized 354 byte log entry
0921 18:44:32.985997 (+444537us) op_driver.cc:444] REPLICATION: finished
0921 18:44:32.986207 (+ 210us) write_op.cc:301] APPLY: starting
0921 18:44:32.986223 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:32.986250 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:32.986263 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:32.986318 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:32.986348 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:32.986356 (+ 8us) write_op.cc:312] APPLY: finished
0921 18:44:32.986416 (+ 60us) log.cc:842] Serialized 22 byte log entry
0921 18:44:32.986506 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:32.986516 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:32.986520 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":593,"prepare.run_wall_time_us":840670,"replication_time_us":444760,"thread_start_us":64,"threads_started":1}]]}
I20260921 18:44:32.987452 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 20) took 1136 ms. Trace:
I20260921 18:44:32.987510 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 21) took 1136 ms. Trace:
I20260921 18:44:32.987571 4955 rpcz_store.cc:276] 0921 18:44:31.851192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:31.851246 (+ 54us) service_pool.cc:227] Handling call
0921 18:44:31.851348 (+ 102us) tablet_service.cc:2780] Created scanner f807c42cf3184e2cb1479a3d8c90be6b for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 1ee79e3bc25d4b449b365b0b97fdaea1
0921 18:44:31.851476 (+ 128us) tablet_service.cc:2920] Creating iterator
0921 18:44:31.851492 (+ 16us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:32.541380 (+689888us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:32.986602 (+445222us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1135102 microseconds
0921 18:44:32.986680 (+ 78us) tablet_service.cc:2945] Iterator created
0921 18:44:32.986953 (+ 273us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:32.986977 (+ 24us) tablet_service.cc:3010] has_more: true
0921 18:44:32.986998 (+ 21us) tablet_service.cc:3027] Continuing scan request
0921 18:44:32.987054 (+ 56us) tablet_service.cc:3091] Found scanner f807c42cf3184e2cb1479a3d8c90be6b for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 1ee79e3bc25d4b449b365b0b97fdaea1
0921 18:44:32.987501 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":70144}
I20260921 18:44:32.987532 4953 rpcz_store.cc:276] 0921 18:44:31.850566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:31.850635 (+ 69us) service_pool.cc:227] Handling call
0921 18:44:31.850798 (+ 163us) tablet_service.cc:2780] Created scanner 4033948f260f45d69cf1b643c4ae5399 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 5cd4ef1a9b2042a9b11e2a26aaa88420
0921 18:44:31.850993 (+ 195us) tablet_service.cc:2920] Creating iterator
0921 18:44:31.851029 (+ 36us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:32.541349 (+690320us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:32.986583 (+445234us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1135540 microseconds
0921 18:44:32.986701 (+ 118us) tablet_service.cc:2945] Iterator created
0921 18:44:32.986953 (+ 252us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:32.986973 (+ 20us) tablet_service.cc:3010] has_more: true
0921 18:44:32.986998 (+ 25us) tablet_service.cc:3027] Continuing scan request
0921 18:44:32.987047 (+ 49us) tablet_service.cc:3091] Found scanner 4033948f260f45d69cf1b643c4ae5399 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 5cd4ef1a9b2042a9b11e2a26aaa88420
0921 18:44:32.987443 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:44:32.988404 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.138s user 0.001s sys 0.000s
W20260921 18:44:32.988405 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.138s user 0.000s sys 0.000s
I20260921 18:44:33.646888 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:33.647199 4863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.130:51975 (request call id 758) took 1172 ms. Trace:
I20260921 18:44:33.647293 4863 rpcz_store.cc:276] 0921 18:44:32.475061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:32.475129 (+ 68us) service_pool.cc:227] Handling call
0921 18:44:32.475254 (+ 125us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:44:32.475322 (+ 68us) raft_consensus.cc:1520] Early marking committed up to index 8
0921 18:44:32.475383 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:44:32.475476 (+ 93us) write_op.cc:270] Start()
0921 18:44:33.152624 (+677148us) write_op.cc:276] Timestamp: P: 1790016270368302 usec, L: 0
0921 18:44:33.152953 (+ 329us) write_op.cc:270] Start()
0921 18:44:33.646118 (+493165us) write_op.cc:276] Timestamp: P: 1790016270948334 usec, L: 0
0921 18:44:33.646460 (+ 342us) log.cc:842] Serialized 708 byte log entry
0921 18:44:33.646590 (+ 130us) raft_consensus.cc:1631] Marking committed up to 10
0921 18:44:33.646801 (+ 211us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:44:33.646834 (+ 33us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:44:33.647141 (+ 307us) raft_consensus.cc:1671] finished
0921 18:44:33.647143 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:44:33.647185 (+ 42us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:33.152913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:33.152998 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:33.153001 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:33.153003 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:33.153080 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:33.153091 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:33.153092 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:33.153125 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:44:33.153126 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:33.646618 (+493492us) op_driver.cc:444] REPLICATION: finished
0921 18:44:33.646758 (+ 140us) write_op.cc:301] APPLY: starting
0921 18:44:33.646774 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:33.646806 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:33.646807 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:33.646905 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:33.646931 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:33.646955 (+ 24us) write_op.cc:312] APPLY: finished
0921 18:44:33.647009 (+ 54us) log.cc:842] Serialized 22 byte log entry
0921 18:44:33.647086 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:33.647095 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:33.647099 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0921 18:44:33.647152 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:33.647154 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0921 18:44:33.646299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:33.646402 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:33.646407 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:33.646408 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:33.646497 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:33.646537 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:33.646542 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:33.646589 (+ 47us) tablet.cc:699] Row locks acquired
0921 18:44:33.646590 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:33.646716 (+ 126us) op_driver.cc:444] REPLICATION: finished
0921 18:44:33.646904 (+ 188us) write_op.cc:301] APPLY: starting
0921 18:44:33.646912 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:33.646926 (+ 14us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:33.646928 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:33.646978 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:33.646993 (+ 15us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:33.647003 (+ 10us) write_op.cc:312] APPLY: finished
0921 18:44:33.647027 (+ 24us) log.cc:842] Serialized 22 byte log entry
0921 18:44:33.647098 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:33.647103 (+ 5us) write_op.cc:454] Released schema lock
0921 18:44:33.647105 (+ 2us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":208,"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":361,"apply.run_wall_time_us":431,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":188,"prepare.run_cpu_time_us":269,"prepare.run_wall_time_us":269,"replication_time_us":1171132,"thread_start_us":142,"threads_started":2}],["op",{"apply.queue_time_us":160,"num_ops":1,"partition_lock_wait_us":31,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":365,"prepare.run_wall_time_us":366,"replication_time_us":493754,"spinlock_wait_cycles":102784,"thread_start_us":54,"threads_started":1}]]}
I20260921 18:44:34.098539 5248 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:34.098685 5248 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:34.099241 5248 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:34.099665 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:34.099670 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:34.100098 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:34.100755 5283 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=10, attempt_no=0}) took 1979 ms. Trace:
I20260921 18:44:34.100826 5283 rpcz_store.cc:276] 0921 18:44:32.120814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:32.120868 (+ 54us) service_pool.cc:227] Handling call
0921 18:44:34.100744 (+1979876us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:32.541582 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:32.541673 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:32.541676 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:32.541678 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:32.541754 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:32.541765 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:32.541767 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:32.541807 (+ 40us) tablet.cc:699] Row locks acquired
0921 18:44:32.541810 (+ 3us) write_op.cc:260] PREPARE: finished
0921 18:44:32.541848 (+ 38us) write_op.cc:270] Start()
0921 18:44:33.242014 (+700166us) write_op.cc:276] Timestamp: P: 1790016272541793 usec, L: 0
0921 18:44:33.242019 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:44:33.242235 (+ 216us) log.cc:842] Serialized 354 byte log entry
0921 18:44:34.099675 (+857440us) op_driver.cc:444] REPLICATION: finished
0921 18:44:34.099881 (+ 206us) write_op.cc:301] APPLY: starting
0921 18:44:34.099895 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:34.099923 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:34.099937 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:34.099991 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:34.100026 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:34.100030 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:34.100577 (+ 547us) log.cc:842] Serialized 22 byte log entry
0921 18:44:34.100672 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:34.100681 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:34.100686 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":157,"num_ops":1,"prepare.queue_time_us":420360,"prepare.run_cpu_time_us":659,"prepare.run_wall_time_us":700743,"raft.queue_time_us":171,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":857635,"thread_start_us":68,"threads_started":1}]]}
W20260921 18:44:34.101832 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.963s user 0.001s sys 0.000s
W20260921 18:44:34.101884 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.963s user 0.001s sys 0.000s
I20260921 18:44:34.541018 5248 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:34.542264 5283 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=11, attempt_no=0}) took 1553 ms. Trace:
I20260921 18:44:34.542346 5283 rpcz_store.cc:276] 0921 18:44:32.988324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:32.988378 (+ 54us) service_pool.cc:227] Handling call
0921 18:44:34.542250 (+1553872us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:33.242359 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:33.242459 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:33.242463 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:33.242464 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:33.242533 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:33.242542 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:33.242545 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:33.242590 (+ 45us) tablet.cc:699] Row locks acquired
0921 18:44:33.242591 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:33.242645 (+ 54us) write_op.cc:270] Start()
0921 18:44:33.435790 (+193145us) write_op.cc:276] Timestamp: P: 1790016273242601 usec, L: 0
0921 18:44:33.435794 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:33.435986 (+ 192us) log.cc:842] Serialized 354 byte log entry
0921 18:44:34.541849 (+1105863us) op_driver.cc:444] REPLICATION: finished
0921 18:44:34.541938 (+ 89us) write_op.cc:301] APPLY: starting
0921 18:44:34.541952 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:34.541979 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:34.542002 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:34.542049 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:34.542072 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:34.542075 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:44:34.542125 (+ 50us) log.cc:842] Serialized 22 byte log entry
0921 18:44:34.542167 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:34.542174 (+ 7us) write_op.cc:454] Released schema lock
0921 18:44:34.542178 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42,"num_ops":1,"prepare.queue_time_us":253635,"prepare.run_cpu_time_us":655,"prepare.run_wall_time_us":193721,"replication_time_us":1106035}]]}
I20260921 18:44:34.837296 5248 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:34.837430 5248 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:34.837953 5248 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:34.838255 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:34.838491 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:34.839761 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
W20260921 18:44:34.839905 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.588s user 0.001s sys 0.000s
W20260921 18:44:34.839939 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.588s user 0.001s sys 0.000s
I20260921 18:44:34.839929 5248 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:35.125748 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:35.125909 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:35.126402 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:35.126713 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:35.126745 4863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.130:51975 (request call id 759) took 1478 ms. Trace:
I20260921 18:44:35.126915 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:35.126816 4863 rpcz_store.cc:276] 0921 18:44:33.648505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:33.648564 (+ 59us) service_pool.cc:227] Handling call
0921 18:44:33.648703 (+ 139us) raft_consensus.cc:1457] Updating replica for 3 ops
0921 18:44:33.648802 (+ 99us) raft_consensus.cc:1520] Early marking committed up to index 0
0921 18:44:33.648804 (+ 2us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0921 18:44:33.648865 (+ 61us) write_op.cc:270] Start()
0921 18:44:34.066061 (+417196us) write_op.cc:276] Timestamp: P: 1790016271701036 usec, L: 0
0921 18:44:34.066328 (+ 267us) write_op.cc:270] Start()
0921 18:44:34.145500 (+ 79172us) write_op.cc:276] Timestamp: P: 1790016272541793 usec, L: 0
0921 18:44:34.145799 (+ 299us) write_op.cc:270] Start()
0921 18:44:35.124981 (+979182us) write_op.cc:276] Timestamp: P: 1790016273242601 usec, L: 0
0921 18:44:35.125425 (+ 444us) log.cc:842] Serialized 1062 byte log entry
0921 18:44:35.125567 (+ 142us) raft_consensus.cc:1631] Marking committed up to 11
0921 18:44:35.125691 (+ 124us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:44:35.125715 (+ 24us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:44:35.126687 (+ 972us) raft_consensus.cc:1671] finished
0921 18:44:35.126689 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:44:35.126735 (+ 46us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:34.066241 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:34.066340 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:34.066347 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:44:34.066350 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:34.066423 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:34.066434 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:34.066441 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:34.066497 (+ 56us) tablet.cc:699] Row locks acquired
0921 18:44:34.066498 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:35.125593 (+1059095us) op_driver.cc:444] REPLICATION: finished
0921 18:44:35.125937 (+ 344us) write_op.cc:301] APPLY: starting
0921 18:44:35.125948 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:35.125974 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:35.125976 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:35.126024 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:35.126054 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:35.126072 (+ 18us) write_op.cc:312] APPLY: finished
0921 18:44:35.126123 (+ 51us) log.cc:842] Serialized 22 byte log entry
0921 18:44:35.126183 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:35.126191 (+ 8us) write_op.cc:454] Released schema lock
0921 18:44:35.126196 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0921 18:44:35.126256 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:35.126261 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0921 18:44:34.145740 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:34.145835 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:34.145839 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:34.145841 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:34.145907 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:34.145915 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:34.145917 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:34.145970 (+ 53us) tablet.cc:699] Row locks acquired
0921 18:44:34.145973 (+ 3us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0921 18:44:35.125275 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:35.125370 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:35.125374 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:35.125376 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:35.125455 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:35.125466 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:35.125493 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:35.125539 (+ 46us) tablet.cc:699] Row locks acquired
0921 18:44:35.125542 (+ 3us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":183,"child_traces":[["op",{"apply.queue_time_us":292,"apply.run_cpu_time_us":372,"apply.run_wall_time_us":372,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":324,"prepare.run_wall_time_us":324,"replication_time_us":1476718,"thread_start_us":51,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":345,"prepare.run_wall_time_us":378}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":171,"prepare.run_cpu_time_us":318,"prepare.run_wall_time_us":318,"thread_start_us":94,"threads_started":1}]]}
I20260921 18:44:35.127357 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:35.127712 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:35.291947 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:35.292078 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:35.292567 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:35.293033 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:35.293095 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:35.293435 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:35.293614 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:35.914090 5290 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:35.914230 5290 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:35.914798 5290 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:35.915109 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:35.915323 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:35.915771 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:35.916484 5293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=13, attempt_no=0}) took 1372 ms. Trace:
I20260921 18:44:35.916554 5293 rpcz_store.cc:276] 0921 18:44:34.543561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:34.543612 (+ 51us) service_pool.cc:227] Handling call
0921 18:44:35.916471 (+1372859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:34.711965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:34.712062 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:34.712066 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:34.712067 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:34.712157 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:34.712167 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:34.712169 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:34.712210 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:34.712211 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:34.712248 (+ 37us) write_op.cc:270] Start()
0921 18:44:35.707378 (+995130us) write_op.cc:276] Timestamp: P: 1790016274712181 usec, L: 0
0921 18:44:35.707381 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:35.707590 (+ 209us) log.cc:842] Serialized 354 byte log entry
0921 18:44:35.915173 (+207583us) op_driver.cc:444] REPLICATION: finished
0921 18:44:35.915400 (+ 227us) write_op.cc:301] APPLY: starting
0921 18:44:35.915417 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:35.915446 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:35.915459 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:35.915511 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:35.915537 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:35.915543 (+ 6us) write_op.cc:312] APPLY: finished
0921 18:44:35.915618 (+ 75us) log.cc:842] Serialized 22 byte log entry
0921 18:44:35.916375 (+ 757us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:35.916385 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:35.916389 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":158,"num_ops":1,"prepare.queue_time_us":168036,"prepare.run_cpu_time_us":620,"prepare.run_wall_time_us":995687,"raft.queue_time_us":39,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":171,"replication_time_us":207645,"thread_start_us":70,"threads_started":1}]]}
I20260921 18:44:35.916795 5290 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:35.916854 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.926s user 0.001s sys 0.000s
W20260921 18:44:35.917312 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.927s user 0.000s sys 0.000s
I20260921 18:44:36.016455 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:36.016618 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:36.017148 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:36.017503 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:36.017547 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:36.017935 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:36.334950 5290 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:36.335103 5290 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:36.335682 5290 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:36.336098 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:36.336206 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:36.336746 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:36.336925 5290 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:36.337574 5293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=14, attempt_no=0}) took 1497 ms. Trace:
I20260921 18:44:36.337658 5293 rpcz_store.cc:276] 0921 18:44:34.840097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:34.840159 (+ 62us) service_pool.cc:227] Handling call
0921 18:44:36.337560 (+1497401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:35.707671 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:35.707802 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:35.707817 (+ 15us) write_op.cc:435] Acquired schema lock
0921 18:44:35.707821 (+ 4us) tablet.cc:660] Decoding operations
0921 18:44:35.707893 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:35.707900 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:35.707902 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:35.707934 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:44:35.707935 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:35.707960 (+ 25us) write_op.cc:270] Start()
0921 18:44:35.985109 (+277149us) write_op.cc:276] Timestamp: P: 1790016275707910 usec, L: 0
0921 18:44:35.985112 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:35.985297 (+ 185us) log.cc:842] Serialized 354 byte log entry
0921 18:44:36.336755 (+351458us) op_driver.cc:444] REPLICATION: finished
0921 18:44:36.337037 (+ 282us) write_op.cc:301] APPLY: starting
0921 18:44:36.337058 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:36.337094 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:36.337113 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:36.337273 (+ 160us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:36.337311 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:36.337319 (+ 8us) write_op.cc:312] APPLY: finished
0921 18:44:36.337389 (+ 70us) log.cc:842] Serialized 22 byte log entry
0921 18:44:36.337451 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:36.337463 (+ 12us) write_op.cc:454] Released schema lock
0921 18:44:36.337488 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":209,"num_ops":1,"prepare.queue_time_us":867249,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":277693,"replication_time_us":351623}]]}
I20260921 18:44:36.705960 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:37.004875 5290 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:37.005030 5290 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:37.005576 5290 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:37.005970 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:37.005970 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:37.006805 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:37.006997 5290 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:37.007216 5300 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=15, attempt_no=0}) took 1088 ms. Trace:
I20260921 18:44:37.007315 5300 rpcz_store.cc:276] 0921 18:44:35.918259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:35.918307 (+ 48us) service_pool.cc:227] Handling call
0921 18:44:37.007206 (+1088899us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:35.985397 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:35.985526 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:35.985531 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:35.985532 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:35.985604 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:35.985612 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:35.985615 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:35.985661 (+ 46us) tablet.cc:699] Row locks acquired
0921 18:44:35.985662 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:35.985696 (+ 34us) write_op.cc:270] Start()
0921 18:44:36.612190 (+626494us) write_op.cc:276] Timestamp: P: 1790016275985652 usec, L: 0
0921 18:44:36.612193 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:36.612371 (+ 178us) log.cc:842] Serialized 354 byte log entry
0921 18:44:37.005783 (+393412us) op_driver.cc:444] REPLICATION: finished
0921 18:44:37.006038 (+ 255us) write_op.cc:301] APPLY: starting
0921 18:44:37.006053 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:37.006082 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:37.006096 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:37.006179 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:37.006207 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:37.006212 (+ 5us) write_op.cc:312] APPLY: finished
0921 18:44:37.006270 (+ 58us) log.cc:842] Serialized 22 byte log entry
0921 18:44:37.007140 (+ 870us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:37.007149 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:37.007154 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":170,"num_ops":1,"prepare.queue_time_us":66788,"prepare.run_cpu_time_us":1241,"prepare.run_wall_time_us":627070,"raft.queue_time_us":47,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"replication_time_us":393569,"thread_start_us":72,"threads_started":1}]]}
W20260921 18:44:37.007570 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.940s user 0.000s sys 0.000s
W20260921 18:44:37.007616 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.940s user 0.001s sys 0.000s
I20260921 18:44:37.428226 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:37.428388 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:37.428864 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:37.429214 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:37.429315 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:37.429934 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:38.013768 5290 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:38.013903 5290 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:38.014375 5290 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:38.014760 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:38.014840 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:38.015300 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:38.015723 5308 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=16, attempt_no=0}) took 1676 ms. Trace:
I20260921 18:44:38.015806 5308 rpcz_store.cc:276] 0921 18:44:36.339214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:36.339256 (+ 42us) service_pool.cc:227] Handling call
0921 18:44:38.015712 (+1676456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:36.612492 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:36.612613 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:36.612617 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:36.612619 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:36.612701 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:36.612708 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:36.612710 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:36.612746 (+ 36us) tablet.cc:699] Row locks acquired
0921 18:44:36.612747 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:36.612775 (+ 28us) write_op.cc:270] Start()
0921 18:44:37.034912 (+422137us) write_op.cc:276] Timestamp: P: 1790016276612675 usec, L: 0
0921 18:44:37.034915 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:37.035058 (+ 143us) log.cc:842] Serialized 354 byte log entry
0921 18:44:38.015043 (+979985us) op_driver.cc:444] REPLICATION: finished
0921 18:44:38.015257 (+ 214us) write_op.cc:301] APPLY: starting
0921 18:44:38.015273 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:38.015301 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:38.015316 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:38.015372 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:38.015400 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:38.015404 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:38.015461 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:44:38.015649 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:38.015658 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:38.015662 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":167,"num_ops":1,"prepare.queue_time_us":272892,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":422639,"replication_time_us":980107,"thread_start_us":60,"threads_started":1}]]}
I20260921 18:44:38.121232 5290 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:38.122663 5308 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=17, attempt_no=0}) took 1114 ms. Trace:
I20260921 18:44:38.122820 5308 rpcz_store.cc:276] 0921 18:44:37.008609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:37.008660 (+ 51us) service_pool.cc:227] Handling call
0921 18:44:38.122650 (+1113990us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:37.035161 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:37.035255 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:37.035260 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:37.035263 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:37.035328 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:37.035335 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:37.035338 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:37.035379 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:37.035381 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:37.035415 (+ 34us) write_op.cc:270] Start()
0921 18:44:37.191559 (+156144us) write_op.cc:276] Timestamp: P: 1790016277035375 usec, L: 0
0921 18:44:37.191562 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:37.191835 (+ 273us) log.cc:842] Serialized 354 byte log entry
0921 18:44:38.122070 (+930235us) op_driver.cc:444] REPLICATION: finished
0921 18:44:38.122187 (+ 117us) write_op.cc:301] APPLY: starting
0921 18:44:38.122201 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:38.122225 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:38.122327 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:38.122381 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:38.122410 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:38.122417 (+ 7us) write_op.cc:312] APPLY: finished
0921 18:44:38.122488 (+ 71us) log.cc:842] Serialized 22 byte log entry
0921 18:44:38.122575 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:38.122585 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:38.122590 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"num_ops":1,"prepare.queue_time_us":26150,"prepare.run_cpu_time_us":677,"prepare.run_wall_time_us":156740,"replication_time_us":930488}]]}
W20260921 18:44:38.123952 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.966s user 0.001s sys 0.000s
W20260921 18:44:38.123952 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.966s user 0.001s sys 0.000s
I20260921 18:44:38.952080 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:38.952513 4863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.130:51975 (request call id 1125) took 1523 ms. Trace:
I20260921 18:44:38.952579 4863 rpcz_store.cc:276] 0921 18:44:37.429367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:37.429420 (+ 53us) service_pool.cc:227] Handling call
0921 18:44:37.429592 (+ 172us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:44:37.429692 (+ 100us) raft_consensus.cc:1520] Early marking committed up to index 17
0921 18:44:37.429848 (+ 156us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:44:37.429949 (+ 101us) write_op.cc:270] Start()
0921 18:44:38.249083 (+819134us) write_op.cc:276] Timestamp: P: 1790016276612675 usec, L: 0
0921 18:44:38.249383 (+ 300us) write_op.cc:270] Start()
0921 18:44:38.951522 (+702139us) write_op.cc:276] Timestamp: P: 1790016277035375 usec, L: 0
0921 18:44:38.951879 (+ 357us) log.cc:842] Serialized 708 byte log entry
0921 18:44:38.951984 (+ 105us) raft_consensus.cc:1631] Marking committed up to 17
0921 18:44:38.951990 (+ 6us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:44:38.952027 (+ 37us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:44:38.952451 (+ 424us) raft_consensus.cc:1671] finished
0921 18:44:38.952453 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:44:38.952500 (+ 47us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:38.249351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:38.249457 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:38.249461 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:38.249463 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:38.249574 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:38.249589 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:38.249593 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:38.249634 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:38.249635 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0921 18:44:38.951816 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:38.951905 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:38.951909 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:38.951910 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:38.951986 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:38.951995 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:38.951996 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:38.952055 (+ 59us) tablet.cc:699] Row locks acquired
0921 18:44:38.952056 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":62,"threads_started":1,"wal-append.queue_time_us":113,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":169,"prepare.run_cpu_time_us":355,"prepare.run_wall_time_us":355,"thread_start_us":88,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":177,"prepare.run_cpu_time_us":326,"prepare.run_wall_time_us":326,"thread_start_us":72,"threads_started":1}]]}
I20260921 18:44:39.382599 5312 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:39.382733 5312 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:39.383276 5312 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:39.383670 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:39.384060 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:39.384519 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:39.385294 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 42) took 1110 ms. Trace:
I20260921 18:44:39.385363 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 43) took 1110 ms. Trace:
I20260921 18:44:39.385509 4953 rpcz_store.cc:276] 0921 18:44:38.275143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:38.275197 (+ 54us) service_pool.cc:227] Handling call
0921 18:44:38.275327 (+ 130us) tablet_service.cc:2780] Created scanner 8d2c1697f98046f79da0645896b6c914 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 629e30f5bb5e49ea863cad7591cc6dc5
0921 18:44:38.275507 (+ 180us) tablet_service.cc:2920] Creating iterator
0921 18:44:38.275529 (+ 22us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:38.856954 (+581425us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:39.384572 (+527618us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1109031 microseconds
0921 18:44:39.384654 (+ 82us) tablet_service.cc:2945] Iterator created
0921 18:44:39.384965 (+ 311us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:39.384984 (+ 19us) tablet_service.cc:3010] has_more: true
0921 18:44:39.385008 (+ 24us) tablet_service.cc:3027] Continuing scan request
0921 18:44:39.385053 (+ 45us) tablet_service.cc:3091] Found scanner 8d2c1697f98046f79da0645896b6c914 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 629e30f5bb5e49ea863cad7591cc6dc5
0921 18:44:39.385355 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":8576}
I20260921 18:44:39.385375 4955 rpcz_store.cc:276] 0921 18:44:38.275087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:38.275123 (+ 36us) service_pool.cc:227] Handling call
0921 18:44:38.275247 (+ 124us) tablet_service.cc:2780] Created scanner 3256f0668fd64cf98b3e6811ad0fb8af for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 62a8e233e20d4d0a8f1e91e697f04705
0921 18:44:38.275414 (+ 167us) tablet_service.cc:2920] Creating iterator
0921 18:44:38.275458 (+ 44us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:38.856922 (+581464us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:39.384563 (+527641us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1109093 microseconds
0921 18:44:39.384641 (+ 78us) tablet_service.cc:2945] Iterator created
0921 18:44:39.384952 (+ 311us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:39.384972 (+ 20us) tablet_service.cc:3010] has_more: true
0921 18:44:39.384995 (+ 23us) tablet_service.cc:3027] Continuing scan request
0921 18:44:39.385036 (+ 41us) tablet_service.cc:3091] Found scanner 3256f0668fd64cf98b3e6811ad0fb8af for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 62a8e233e20d4d0a8f1e91e697f04705
0921 18:44:39.385286 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:44:39.386070 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.112s user 0.001s sys 0.000s
W20260921 18:44:39.386070 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.112s user 0.001s sys 0.000s
I20260921 18:44:39.386127 5320 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=18, attempt_no=0}) took 1369 ms. Trace:
I20260921 18:44:39.386186 5320 rpcz_store.cc:276] 0921 18:44:38.016974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:38.017032 (+ 58us) service_pool.cc:227] Handling call
0921 18:44:39.386115 (+1369083us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:38.017449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:38.017563 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:38.017566 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:38.017567 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:38.017617 (+ 50us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:38.017622 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:38.017624 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:38.017648 (+ 24us) tablet.cc:699] Row locks acquired
0921 18:44:38.017649 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:38.017672 (+ 23us) write_op.cc:270] Start()
0921 18:44:38.856796 (+839124us) write_op.cc:276] Timestamp: P: 1790016278017626 usec, L: 0
0921 18:44:38.856799 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:38.857051 (+ 252us) log.cc:842] Serialized 354 byte log entry
0921 18:44:39.383536 (+526485us) op_driver.cc:444] REPLICATION: finished
0921 18:44:39.383751 (+ 215us) write_op.cc:301] APPLY: starting
0921 18:44:39.383767 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:39.383795 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:39.383805 (+ 10us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:39.384329 (+ 524us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:39.384366 (+ 37us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:39.384370 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:39.384433 (+ 63us) log.cc:842] Serialized 22 byte log entry
0921 18:44:39.386003 (+ 1570us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:39.386011 (+ 8us) write_op.cc:454] Released schema lock
0921 18:44:39.386016 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"num_ops":1,"prepare.queue_time_us":141,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":839690,"raft.queue_time_us":45,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"replication_time_us":526715,"spinlock_wait_cycles":3328,"thread_start_us":134,"threads_started":2}]]}
I20260921 18:44:39.852849 5222 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:39.853000 5222 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:39.853559 5222 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:39.853957 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:39.853967 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:39.854444 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:40.130052 5312 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:40.131405 5325 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=19, attempt_no=0}) took 2007 ms. Trace:
I20260921 18:44:40.131508 5325 rpcz_store.cc:276] 0921 18:44:38.124295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:38.124347 (+ 52us) service_pool.cc:227] Handling call
0921 18:44:40.131389 (+2007042us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:38.857175 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:38.857273 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:38.857278 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:38.857279 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:38.857349 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:38.857359 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:38.857362 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:38.857417 (+ 55us) tablet.cc:699] Row locks acquired
0921 18:44:38.857418 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:38.857445 (+ 27us) write_op.cc:270] Start()
0921 18:44:39.343575 (+486130us) write_op.cc:276] Timestamp: P: 1790016278857423 usec, L: 0
0921 18:44:39.343578 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:39.343765 (+ 187us) log.cc:842] Serialized 354 byte log entry
0921 18:44:40.130842 (+787077us) op_driver.cc:444] REPLICATION: finished
0921 18:44:40.131062 (+ 220us) write_op.cc:301] APPLY: starting
0921 18:44:40.131076 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:40.131100 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:40.131114 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:40.131161 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:40.131183 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:40.131186 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:44:40.131235 (+ 49us) log.cc:842] Serialized 22 byte log entry
0921 18:44:40.131312 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:40.131322 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:40.131327 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":177,"num_ops":1,"prepare.queue_time_us":732510,"prepare.run_cpu_time_us":600,"prepare.run_wall_time_us":486665,"replication_time_us":787241,"thread_start_us":62,"threads_started":1}]]}
I20260921 18:44:40.269533 5222 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:40.272387 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.736s user 0.001s sys 0.000s
W20260921 18:44:40.272584 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.736s user 0.001s sys 0.000s
I20260921 18:44:40.909385 5312 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:40.909552 5312 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:40.910172 5312 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:40.910513 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:40.910588 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:40.911006 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:41.500924 5312 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:41.503355 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 50) took 1079 ms. Trace:
I20260921 18:44:41.503412 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 49) took 1079 ms. Trace:
I20260921 18:44:41.503412 5335 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=21, attempt_no=0}) took 1370 ms. Trace:
I20260921 18:44:41.503451 4953 rpcz_store.cc:276] 0921 18:44:40.423695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:40.423750 (+ 55us) service_pool.cc:227] Handling call
0921 18:44:40.423855 (+ 105us) tablet_service.cc:2780] Created scanner 84469916f39241e0842dab36952f045d for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 32d39577dc3e4a38b4eacfe734d68f90
0921 18:44:40.424050 (+ 195us) tablet_service.cc:2920] Creating iterator
0921 18:44:40.424069 (+ 19us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:40.818816 (+394747us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:41.502577 (+683761us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1078499 microseconds
0921 18:44:41.502655 (+ 78us) tablet_service.cc:2945] Iterator created
0921 18:44:41.502972 (+ 317us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:41.503003 (+ 31us) tablet_service.cc:3010] has_more: true
0921 18:44:41.503040 (+ 37us) tablet_service.cc:3027] Continuing scan request
0921 18:44:41.503081 (+ 41us) tablet_service.cc:3091] Found scanner 84469916f39241e0842dab36952f045d for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 32d39577dc3e4a38b4eacfe734d68f90
0921 18:44:41.503345 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":896}
I20260921 18:44:41.503616 5335 rpcz_store.cc:276] 0921 18:44:40.132771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:40.132827 (+ 56us) service_pool.cc:227] Handling call
0921 18:44:41.503401 (+1370574us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:40.133274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:40.133352 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:40.133354 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:44:40.133355 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:40.133422 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:40.133427 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:40.133429 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:40.133461 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:44:40.133462 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:40.133527 (+ 65us) write_op.cc:270] Start()
0921 18:44:40.818670 (+685143us) write_op.cc:276] Timestamp: P: 1790016280133474 usec, L: 0
0921 18:44:40.818672 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:44:40.818909 (+ 237us) log.cc:842] Serialized 354 byte log entry
0921 18:44:41.502025 (+683116us) op_driver.cc:444] REPLICATION: finished
0921 18:44:41.502248 (+ 223us) write_op.cc:301] APPLY: starting
0921 18:44:41.502265 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:41.502294 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:41.502309 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:41.502363 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:41.502395 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:41.502399 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:41.502460 (+ 61us) log.cc:842] Serialized 22 byte log entry
0921 18:44:41.503330 (+ 870us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:41.503339 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:41.503344 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":160,"num_ops":1,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":685694,"replication_time_us":683327,"thread_start_us":145,"threads_started":2}]]}
W20260921 18:44:41.504199 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.081s user 0.001s sys 0.000s
I20260921 18:44:41.503531 4955 rpcz_store.cc:276] 0921 18:44:40.423612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:40.423655 (+ 43us) service_pool.cc:227] Handling call
0921 18:44:40.423803 (+ 148us) tablet_service.cc:2780] Created scanner a6be79b102ea4250817a66f66c8f7f84 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is cf97a7386e524f9f830d5d99f1425252
0921 18:44:40.423967 (+ 164us) tablet_service.cc:2920] Creating iterator
0921 18:44:40.423989 (+ 22us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:40.818785 (+394796us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:41.502595 (+683810us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1078597 microseconds
0921 18:44:41.502670 (+ 75us) tablet_service.cc:2945] Iterator created
0921 18:44:41.502971 (+ 301us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:41.502993 (+ 22us) tablet_service.cc:3010] has_more: true
0921 18:44:41.503024 (+ 31us) tablet_service.cc:3027] Continuing scan request
0921 18:44:41.503070 (+ 46us) tablet_service.cc:3091] Found scanner a6be79b102ea4250817a66f66c8f7f84 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is cf97a7386e524f9f830d5d99f1425252
0921 18:44:41.503403 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:44:41.504714 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.082s user 0.001s sys 0.000s
I20260921 18:44:41.763384 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:41.763581 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:41.764241 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:41.764647 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:41.765203 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:41.765579 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:41.820866 5312 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:41.821043 5312 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:41.821576 5312 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:41.821992 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:41.822089 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:41.822261 5335 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=22, attempt_no=0}) took 1549 ms. Trace:
I20260921 18:44:41.822331 5335 rpcz_store.cc:276] 0921 18:44:40.272603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:40.272670 (+ 67us) service_pool.cc:227] Handling call
0921 18:44:41.822248 (+1549578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:40.819004 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:40.819100 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:40.819106 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:40.819107 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:40.819192 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:40.819199 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:40.819201 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:40.819256 (+ 55us) tablet.cc:699] Row locks acquired
0921 18:44:40.819258 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:40.819291 (+ 33us) write_op.cc:270] Start()
0921 18:44:41.721420 (+902129us) write_op.cc:276] Timestamp: P: 1790016280819233 usec, L: 0
0921 18:44:41.721422 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:44:41.721711 (+ 289us) log.cc:842] Serialized 354 byte log entry
0921 18:44:41.821755 (+100044us) op_driver.cc:444] REPLICATION: finished
0921 18:44:41.821873 (+ 118us) write_op.cc:301] APPLY: starting
0921 18:44:41.821891 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:41.821919 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:41.821938 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:41.822005 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:41.822034 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:41.822038 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:41.822103 (+ 65us) log.cc:842] Serialized 22 byte log entry
0921 18:44:41.822175 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:41.822185 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:41.822189 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"num_ops":1,"prepare.queue_time_us":545990,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":902794,"raft.queue_time_us":42,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":100309,"spinlock_wait_cycles":4480}]]}
I20260921 18:44:41.823728 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:41.823977 5312 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:42.558996 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:42.870254 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:42.870415 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:42.870954 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:42.871244 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:42.871361 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:42.871819 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:42.871973 5344 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=23, attempt_no=0}) took 1367 ms. Trace:
I20260921 18:44:42.872035 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:42.872035 5344 rpcz_store.cc:276] 0921 18:44:41.504870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:41.504910 (+ 40us) service_pool.cc:227] Handling call
0921 18:44:42.871961 (+1367051us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:41.721821 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:41.721926 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:41.721932 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:41.721934 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:41.722018 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:41.722025 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:41.722027 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:41.722059 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:44:41.722060 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:41.722087 (+ 27us) write_op.cc:270] Start()
0921 18:44:42.466221 (+744134us) write_op.cc:276] Timestamp: P: 1790016281722035 usec, L: 0
0921 18:44:42.466224 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:42.466431 (+ 207us) log.cc:842] Serialized 354 byte log entry
0921 18:44:42.871299 (+404868us) op_driver.cc:444] REPLICATION: finished
0921 18:44:42.871525 (+ 226us) write_op.cc:301] APPLY: starting
0921 18:44:42.871542 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:42.871569 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:42.871582 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:42.871667 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:42.871701 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:42.871728 (+ 27us) write_op.cc:312] APPLY: finished
0921 18:44:42.871788 (+ 60us) log.cc:842] Serialized 22 byte log entry
0921 18:44:42.871856 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:42.871865 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:42.871901 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":179,"num_ops":1,"prepare.queue_time_us":216650,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":744696,"raft.queue_time_us":38,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":168,"replication_time_us":405055,"thread_start_us":71,"threads_started":1}]]}
W20260921 18:44:42.873428 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.900s user 0.001s sys 0.000s
W20260921 18:44:42.873706 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.900s user 0.000s sys 0.000s
I20260921 18:44:42.953950 5340 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:42.954080 5340 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:42.954607 5340 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:42.954852 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:42.954990 4863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:42.955343 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:42.955622 5340 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:43.975623 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:43.975785 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:43.976285 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:43.976677 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:43.976799 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:43.977214 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:43.977351 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:43.977653 5352 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=24, attempt_no=0}) took 2153 ms. Trace:
I20260921 18:44:43.977764 5352 rpcz_store.cc:276] 0921 18:44:41.823761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:41.823800 (+ 39us) service_pool.cc:227] Handling call
0921 18:44:43.977636 (+2153836us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:42.466544 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:42.466639 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:42.466645 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:42.466648 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:42.466720 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:42.466729 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:42.466732 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:42.466776 (+ 44us) tablet.cc:699] Row locks acquired
0921 18:44:42.466777 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:42.466803 (+ 26us) write_op.cc:270] Start()
0921 18:44:43.343941 (+877138us) write_op.cc:276] Timestamp: P: 1790016282466714 usec, L: 0
0921 18:44:43.343944 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:43.344163 (+ 219us) log.cc:842] Serialized 354 byte log entry
0921 18:44:43.976888 (+632725us) op_driver.cc:444] REPLICATION: finished
0921 18:44:43.977241 (+ 353us) write_op.cc:301] APPLY: starting
0921 18:44:43.977254 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:43.977280 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:43.977294 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:43.977346 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:43.977371 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:43.977375 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:43.977443 (+ 68us) log.cc:842] Serialized 22 byte log entry
0921 18:44:43.977553 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:43.977563 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:43.977568 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":301,"num_ops":1,"prepare.queue_time_us":642471,"prepare.run_cpu_time_us":608,"prepare.run_wall_time_us":877682,"replication_time_us":632923,"thread_start_us":64,"threads_started":1}]]}
W20260921 18:44:43.979202 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.955s user 0.001s sys 0.000s
W20260921 18:44:43.979212 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.955s user 0.000s sys 0.000s
I20260921 18:44:44.045328 5340 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:44.045504 5340 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:44.046036 5340 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:44.046425 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:44.046458 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:44.046876 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:44.900491 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:44.900635 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:44.901158 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:44.901559 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:44.901712 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:44.902132 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:44.902462 5360 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=25, attempt_no=0}) took 2028 ms. Trace:
I20260921 18:44:44.902541 5360 rpcz_store.cc:276] 0921 18:44:42.873793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:42.873842 (+ 49us) service_pool.cc:227] Handling call
0921 18:44:44.902444 (+2028602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:43.344255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:43.344350 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:43.344354 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:43.344356 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:43.344425 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:43.344434 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:43.344436 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:43.344477 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:43.344479 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:43.344511 (+ 32us) write_op.cc:270] Start()
0921 18:44:43.904651 (+560140us) write_op.cc:276] Timestamp: P: 1790016283344455 usec, L: 0
0921 18:44:43.904655 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:43.904853 (+ 198us) log.cc:842] Serialized 354 byte log entry
0921 18:44:44.901790 (+996937us) op_driver.cc:444] REPLICATION: finished
0921 18:44:44.901974 (+ 184us) write_op.cc:301] APPLY: starting
0921 18:44:44.902102 (+ 128us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:44.902138 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:44.902150 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:44.902197 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:44.902250 (+ 53us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:44.902254 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:44.902316 (+ 62us) log.cc:842] Serialized 22 byte log entry
0921 18:44:44.902370 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:44.902380 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:44.902384 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":129,"num_ops":1,"prepare.queue_time_us":470110,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":560673,"replication_time_us":997113,"thread_start_us":65,"threads_started":1}]]}
I20260921 18:44:44.920694 5340 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:45.258668 5340 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:45.258839 5340 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:45.259378 5340 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:45.259721 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:45.259825 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:45.260087 5360 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=26, attempt_no=0}) took 1281 ms. Trace:
I20260921 18:44:45.260165 5360 rpcz_store.cc:276] 0921 18:44:43.979101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:43.979164 (+ 63us) service_pool.cc:227] Handling call
0921 18:44:45.260075 (+1280911us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:43.979547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:43.979602 (+ 55us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:43.979604 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:44:43.979605 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:43.979649 (+ 44us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:43.979654 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:43.979656 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:43.979694 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:44:43.979695 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:43.979716 (+ 21us) write_op.cc:270] Start()
0921 18:44:44.309897 (+330181us) write_op.cc:276] Timestamp: P: 1790016283979692 usec, L: 0
0921 18:44:44.309900 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:44.310127 (+ 227us) log.cc:842] Serialized 354 byte log entry
0921 18:44:45.259647 (+949520us) op_driver.cc:444] REPLICATION: finished
0921 18:44:45.259734 (+ 87us) write_op.cc:301] APPLY: starting
0921 18:44:45.259750 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:45.259777 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:45.259789 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:45.259840 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:45.259865 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:45.259869 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:45.259925 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:44:45.259997 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:45.260007 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:45.260011 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":330629,"replication_time_us":949724}]]}
I20260921 18:44:45.261101 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 62) took 1130 ms. Trace:
I20260921 18:44:45.261225 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 61) took 1130 ms. Trace:
I20260921 18:44:45.261365 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:45.261296 4955 rpcz_store.cc:276] 0921 18:44:44.130372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:44.130435 (+ 63us) service_pool.cc:227] Handling call
0921 18:44:44.130636 (+ 201us) tablet_service.cc:2780] Created scanner 5a82ae6f7f314385bf1645e3a65dd9ee for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is b4fefd69886244698e9084ebafb1c57d
0921 18:44:44.130839 (+ 203us) tablet_service.cc:2920] Creating iterator
0921 18:44:44.130871 (+ 32us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:44.310050 (+179179us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:45.260289 (+950239us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1129412 microseconds
0921 18:44:45.260339 (+ 50us) tablet_service.cc:2945] Iterator created
0921 18:44:45.260643 (+ 304us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:45.260663 (+ 20us) tablet_service.cc:3010] has_more: true
0921 18:44:45.260691 (+ 28us) tablet_service.cc:3027] Continuing scan request
0921 18:44:45.260738 (+ 47us) tablet_service.cc:3091] Found scanner 5a82ae6f7f314385bf1645e3a65dd9ee for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is b4fefd69886244698e9084ebafb1c57d
0921 18:44:45.261220 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":21376}
I20260921 18:44:45.261526 5340 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:45.261175 4953 rpcz_store.cc:276] 0921 18:44:44.130427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:44.130480 (+ 53us) service_pool.cc:227] Handling call
0921 18:44:44.130636 (+ 156us) tablet_service.cc:2780] Created scanner c9667febca724ead9cdd2868693ce05b for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 0e83036be16845bba707ecf138cac55a
0921 18:44:44.130838 (+ 202us) tablet_service.cc:2920] Creating iterator
0921 18:44:44.130873 (+ 35us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:44.310050 (+179177us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:45.260250 (+950200us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1129366 microseconds
0921 18:44:45.260339 (+ 89us) tablet_service.cc:2945] Iterator created
0921 18:44:45.260643 (+ 304us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:45.260667 (+ 24us) tablet_service.cc:3010] has_more: true
0921 18:44:45.260691 (+ 24us) tablet_service.cc:3027] Continuing scan request
0921 18:44:45.260738 (+ 47us) tablet_service.cc:3091] Found scanner c9667febca724ead9cdd2868693ce05b for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 0e83036be16845bba707ecf138cac55a
0921 18:44:45.261087 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":88704}
W20260921 18:44:45.262045 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.132s user 0.001s sys 0.000s
W20260921 18:44:45.262077 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.132s user 0.000s sys 0.001s
I20260921 18:44:45.595811 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:46.018942 5340 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:46.019097 5340 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:46.019526 5340 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:46.020002 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:46.020090 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:46.020581 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:46.020900 5340 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:46.021211 5368 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=27, attempt_no=0}) took 1117 ms. Trace:
I20260921 18:44:46.021303 5368 rpcz_store.cc:276] 0921 18:44:44.903855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:44.903902 (+ 47us) service_pool.cc:227] Handling call
0921 18:44:46.021199 (+1117297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:44.904312 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:44.904378 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:44.904380 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:44:44.904382 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:44.904430 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:44.904435 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:44.904437 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:44.904462 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:44:44.904462 (+ 0us) write_op.cc:260] PREPARE: finished
0921 18:44:44.904485 (+ 23us) write_op.cc:270] Start()
0921 18:44:45.688649 (+784164us) write_op.cc:276] Timestamp: P: 1790016284904425 usec, L: 0
0921 18:44:45.688652 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:45.688858 (+ 206us) log.cc:842] Serialized 354 byte log entry
0921 18:44:46.020145 (+331287us) op_driver.cc:444] REPLICATION: finished
0921 18:44:46.020484 (+ 339us) write_op.cc:301] APPLY: starting
0921 18:44:46.020500 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:46.020529 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:46.020544 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:46.020595 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:46.020623 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:46.020627 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:46.020683 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:44:46.021123 (+ 440us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:46.021132 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:46.021137 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":290,"num_ops":1,"prepare.queue_time_us":138,"prepare.run_cpu_time_us":522,"prepare.run_wall_time_us":784592,"replication_time_us":331473,"thread_start_us":121,"threads_started":2}]]}
W20260921 18:44:46.022812 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.610s user 0.000s sys 0.001s
W20260921 18:44:46.022812 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.610s user 0.001s sys 0.000s
I20260921 18:44:46.646137 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:46.646289 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:46.646811 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:46.647135 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:46.647306 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:46.647703 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:47.181409 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:47.181579 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:47.182144 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:47.182492 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:47.182539 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:47.182966 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:47.184216 5378 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=28, attempt_no=0}) took 1922 ms. Trace:
I20260921 18:44:47.184289 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 68) took 1010 ms. Trace:
I20260921 18:44:47.184432 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 69) took 1010 ms. Trace:
I20260921 18:44:47.184333 4953 rpcz_store.cc:276] 0921 18:44:46.173892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:46.173933 (+ 41us) service_pool.cc:227] Handling call
0921 18:44:46.174126 (+ 193us) tablet_service.cc:2780] Created scanner b006bb0240aa40ea93e486c28c6d60ef for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 25a6dd2f997343b18c52fde4e5e245f3
0921 18:44:46.174339 (+ 213us) tablet_service.cc:2920] Creating iterator
0921 18:44:46.174373 (+ 34us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:46.555494 (+381121us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:47.183479 (+627985us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1009096 microseconds
0921 18:44:47.183551 (+ 72us) tablet_service.cc:2945] Iterator created
0921 18:44:47.183841 (+ 290us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:47.183867 (+ 26us) tablet_service.cc:3010] has_more: true
0921 18:44:47.183889 (+ 22us) tablet_service.cc:3027] Continuing scan request
0921 18:44:47.183942 (+ 53us) tablet_service.cc:3091] Found scanner b006bb0240aa40ea93e486c28c6d60ef for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 25a6dd2f997343b18c52fde4e5e245f3
0921 18:44:47.184282 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":38784}
I20260921 18:44:47.184516 4955 rpcz_store.cc:276] 0921 18:44:46.173980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:46.174014 (+ 34us) service_pool.cc:227] Handling call
0921 18:44:46.174141 (+ 127us) tablet_service.cc:2780] Created scanner 8391aef8cde64503bfba055f0a5e22c1 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 6b7f909667a144bbbcd8c244b7f66d0f
0921 18:44:46.174327 (+ 186us) tablet_service.cc:2920] Creating iterator
0921 18:44:46.174376 (+ 49us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:46.556006 (+381630us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:47.183479 (+627473us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1009093 microseconds
0921 18:44:47.183551 (+ 72us) tablet_service.cc:2945] Iterator created
0921 18:44:47.183841 (+ 290us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:47.183863 (+ 22us) tablet_service.cc:3010] has_more: true
0921 18:44:47.183889 (+ 26us) tablet_service.cc:3027] Continuing scan request
0921 18:44:47.183934 (+ 45us) tablet_service.cc:3091] Found scanner 8391aef8cde64503bfba055f0a5e22c1 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 6b7f909667a144bbbcd8c244b7f66d0f
0921 18:44:47.184423 (+ 489us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":29312}
I20260921 18:44:47.184304 5378 rpcz_store.cc:276] 0921 18:44:45.261748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:45.261783 (+ 35us) service_pool.cc:227] Handling call
0921 18:44:47.184202 (+1922419us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:45.688940 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:45.689040 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:45.689044 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:45.689045 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:45.689122 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:45.689130 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:45.689133 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:45.689171 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:44:45.689172 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:45.689208 (+ 36us) write_op.cc:270] Start()
0921 18:44:46.555364 (+866156us) write_op.cc:276] Timestamp: P: 1790016285689153 usec, L: 0
0921 18:44:46.555368 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:46.555615 (+ 247us) log.cc:842] Serialized 354 byte log entry
0921 18:44:47.182792 (+627177us) op_driver.cc:444] REPLICATION: finished
0921 18:44:47.183037 (+ 245us) write_op.cc:301] APPLY: starting
0921 18:44:47.183054 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:47.183083 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:47.183096 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:47.183276 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:47.183304 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:47.183308 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:47.183370 (+ 62us) log.cc:842] Serialized 22 byte log entry
0921 18:44:47.184106 (+ 736us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:47.184115 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:47.184120 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":192,"num_ops":1,"prepare.queue_time_us":426853,"prepare.run_cpu_time_us":648,"prepare.run_wall_time_us":866730,"replication_time_us":627401,"spinlock_wait_cycles":16640,"thread_start_us":63,"threads_started":1}]]}
W20260921 18:44:47.185000 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.012s user 0.000s sys 0.000s
W20260921 18:44:47.185032 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.012s user 0.001s sys 0.000s
I20260921 18:44:47.572878 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:47.574234 5378 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=29, attempt_no=0}) took 1551 ms. Trace:
I20260921 18:44:47.574309 5378 rpcz_store.cc:276] 0921 18:44:46.022782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:46.023020 (+ 238us) service_pool.cc:227] Handling call
0921 18:44:47.574220 (+1551200us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:46.555695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:46.555795 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:46.555799 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:46.555800 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:46.555872 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:46.555880 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:46.555882 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:46.555931 (+ 49us) tablet.cc:699] Row locks acquired
0921 18:44:46.555932 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:46.555961 (+ 29us) write_op.cc:270] Start()
0921 18:44:46.740121 (+184160us) write_op.cc:276] Timestamp: P: 1790016286555937 usec, L: 0
0921 18:44:46.740125 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:46.740300 (+ 175us) log.cc:842] Serialized 354 byte log entry
0921 18:44:47.573705 (+833405us) op_driver.cc:444] REPLICATION: finished
0921 18:44:47.573799 (+ 94us) write_op.cc:301] APPLY: starting
0921 18:44:47.573813 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:47.573836 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:47.573900 (+ 64us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:47.573983 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:47.574026 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:47.574030 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:47.574080 (+ 50us) log.cc:842] Serialized 22 byte log entry
0921 18:44:47.574145 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:47.574155 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:47.574160 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":532369,"prepare.run_cpu_time_us":603,"prepare.run_wall_time_us":184679,"replication_time_us":833560}]]}
I20260921 18:44:47.584544 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:47.969206 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:47.969341 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:47.969892 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:47.970329 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:47.970439 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:47.971149 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
W20260921 18:44:47.971789 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.636s user 0.000s sys 0.001s
W20260921 18:44:47.971799 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.636s user 0.001s sys 0.000s
I20260921 18:44:48.364741 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:48.550222 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:48.550334 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:48.550782 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:48.551074 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:48.551537 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:48.551846 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:48.552048 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:48.896373 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:48.896535 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:48.897097 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:48.897445 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:48.897505 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:48.897904 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:48.898105 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:48.899346 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.777s user 0.001s sys 0.000s
W20260921 18:44:48.899381 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.777s user 0.000s sys 0.001s
I20260921 18:44:49.513301 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:49.513422 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:49.513922 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:49.514433 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:49.514607 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:49.514979 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:49.670665 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:49.670811 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:49.671274 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:49.671730 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:49.671730 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:49.672173 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:49.672439 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:49.673032 5391 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=33, attempt_no=0}) took 1305 ms. Trace:
I20260921 18:44:49.673106 5391 rpcz_store.cc:276] 0921 18:44:48.367067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:48.367124 (+ 57us) service_pool.cc:227] Handling call
0921 18:44:49.673018 (+1305894us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:48.799284 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:48.799415 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:48.799420 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:48.799421 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:48.799495 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:48.799508 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:48.799510 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:48.799564 (+ 54us) tablet.cc:699] Row locks acquired
0921 18:44:48.799566 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:48.799600 (+ 34us) write_op.cc:270] Start()
0921 18:44:49.026726 (+227126us) write_op.cc:276] Timestamp: P: 1790016288799558 usec, L: 0
0921 18:44:49.026730 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:49.026955 (+ 225us) log.cc:842] Serialized 354 byte log entry
0921 18:44:49.672392 (+645437us) op_driver.cc:444] REPLICATION: finished
0921 18:44:49.672637 (+ 245us) write_op.cc:301] APPLY: starting
0921 18:44:49.672650 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:49.672674 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:49.672714 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:49.672779 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:49.672805 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:49.672809 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:49.672866 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:44:49.672919 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:49.672930 (+ 11us) write_op.cc:454] Released schema lock
0921 18:44:49.672935 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":1,"prepare.queue_time_us":431906,"prepare.run_cpu_time_us":688,"prepare.run_wall_time_us":227745,"replication_time_us":645643,"thread_start_us":93,"threads_started":1}]]}
I20260921 18:44:50.688438 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:50.688594 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:50.689081 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:50.689536 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:50.689616 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:50.690104 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:50.690300 5398 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=34, attempt_no=0}) took 1791 ms. Trace:
I20260921 18:44:50.690358 5398 rpcz_store.cc:276] 0921 18:44:48.899236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:48.899277 (+ 41us) service_pool.cc:227] Handling call
0921 18:44:50.690287 (+1791010us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:49.027086 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:49.027188 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:49.027193 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:49.027195 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:49.027283 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:49.027291 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:49.027294 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:49.027330 (+ 36us) tablet.cc:699] Row locks acquired
0921 18:44:49.027331 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:49.027390 (+ 59us) write_op.cc:270] Start()
0921 18:44:49.901547 (+874157us) write_op.cc:276] Timestamp: P: 1790016289027326 usec, L: 0
0921 18:44:49.901550 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:49.901770 (+ 220us) log.cc:842] Serialized 354 byte log entry
0921 18:44:50.689693 (+787923us) op_driver.cc:444] REPLICATION: finished
0921 18:44:50.689920 (+ 227us) write_op.cc:301] APPLY: starting
0921 18:44:50.689935 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:50.689965 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:50.689983 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:50.690037 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:50.690066 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:50.690070 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:50.690128 (+ 58us) log.cc:842] Serialized 22 byte log entry
0921 18:44:50.690207 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:50.690217 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:50.690222 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":184,"num_ops":1,"prepare.queue_time_us":127478,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":874747,"replication_time_us":788125,"thread_start_us":47,"threads_started":1}]]}
I20260921 18:44:50.690831 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:50.691191 4955 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 80) took 1640 ms. Trace:
I20260921 18:44:50.691253 4955 rpcz_store.cc:276] 0921 18:44:49.050602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:49.050654 (+ 52us) service_pool.cc:227] Handling call
0921 18:44:49.050760 (+ 106us) tablet_service.cc:2780] Created scanner 2eb7fcf0189641ca957b4af77a3fe9ef for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 52eb9458272d4ae4a93a179b9f8a76e9
0921 18:44:49.050904 (+ 144us) tablet_service.cc:2920] Creating iterator
0921 18:44:49.050926 (+ 22us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:49.901647 (+850721us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:50.690312 (+788665us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1639367 microseconds
0921 18:44:50.690401 (+ 89us) tablet_service.cc:2945] Iterator created
0921 18:44:50.690686 (+ 285us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:50.690708 (+ 22us) tablet_service.cc:3010] has_more: true
0921 18:44:50.690737 (+ 29us) tablet_service.cc:3027] Continuing scan request
0921 18:44:50.690787 (+ 50us) tablet_service.cc:3091] Found scanner 2eb7fcf0189641ca957b4af77a3fe9ef for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 52eb9458272d4ae4a93a179b9f8a76e9
0921 18:44:50.691184 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":12672}
I20260921 18:44:50.691506 4953 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:59814 (request call id 79) took 1641 ms. Trace:
I20260921 18:44:50.691576 4953 rpcz_store.cc:276] 0921 18:44:49.050495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:49.050560 (+ 65us) service_pool.cc:227] Handling call
0921 18:44:49.050736 (+ 176us) tablet_service.cc:2780] Created scanner cf43204e21b1494fbe4d0ad5260b7b45 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 452cf1ec54434f6bbfdd282cac1f00b2
0921 18:44:49.050919 (+ 183us) tablet_service.cc:2920] Creating iterator
0921 18:44:49.050948 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:44:49.901667 (+850719us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:44:50.690294 (+788627us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1639335 microseconds
0921 18:44:50.690383 (+ 89us) tablet_service.cc:2945] Iterator created
0921 18:44:50.690702 (+ 319us) tablet_service.cc:2967] Iterator init: OK
0921 18:44:50.690724 (+ 22us) tablet_service.cc:3010] has_more: true
0921 18:44:50.690749 (+ 25us) tablet_service.cc:3027] Continuing scan request
0921 18:44:50.690791 (+ 42us) tablet_service.cc:3091] Found scanner cf43204e21b1494fbe4d0ad5260b7b45 for tablet 596bd65c0a4d44f5a1cc6390db4f0757, query id is 452cf1ec54434f6bbfdd282cac1f00b2
0921 18:44:50.691497 (+ 706us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":3328}
W20260921 18:44:50.692188 5219 scanner-internal.cc:458] Time spent opening tablet: real 1.642s user 0.001s sys 0.000s
W20260921 18:44:50.692190 5220 scanner-internal.cc:458] Time spent opening tablet: real 1.642s user 0.000s sys 0.000s
I20260921 18:44:51.085076 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:51.085243 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:51.085819 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:51.086141 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:51.086228 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:51.086638 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:51.086932 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:51.087193 5398 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=35, attempt_no=0}) took 1412 ms. Trace:
I20260921 18:44:51.087275 5398 rpcz_store.cc:276] 0921 18:44:49.674677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:49.674730 (+ 53us) service_pool.cc:227] Handling call
0921 18:44:51.087185 (+1412455us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:49.901872 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:49.901980 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:49.901984 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:49.901986 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:49.902076 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:49.902085 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:49.902087 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:49.902127 (+ 40us) tablet.cc:699] Row locks acquired
0921 18:44:49.902129 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:49.902160 (+ 31us) write_op.cc:270] Start()
0921 18:44:50.707303 (+805143us) write_op.cc:276] Timestamp: P: 1790016289902102 usec, L: 0
0921 18:44:50.707307 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:50.707458 (+ 151us) log.cc:842] Serialized 354 byte log entry
0921 18:44:51.085906 (+378448us) op_driver.cc:444] REPLICATION: finished
0921 18:44:51.086043 (+ 137us) write_op.cc:301] APPLY: starting
0921 18:44:51.086059 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:51.086085 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:51.086097 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:51.086769 (+ 672us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:51.087019 (+ 250us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:51.087025 (+ 6us) write_op.cc:312] APPLY: finished
0921 18:44:51.087079 (+ 54us) log.cc:842] Serialized 22 byte log entry
0921 18:44:51.087130 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:51.087137 (+ 7us) write_op.cc:454] Released schema lock
0921 18:44:51.087140 (+ 3us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"num_ops":1,"prepare.queue_time_us":226840,"prepare.run_cpu_time_us":565,"prepare.run_wall_time_us":805653,"replication_time_us":378578}]]}
I20260921 18:44:51.132627 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:51.132848 4863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.130:51975 (request call id 2113) took 1618 ms. Trace:
I20260921 18:44:51.132925 4863 rpcz_store.cc:276] 0921 18:44:49.514865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:49.514909 (+ 44us) service_pool.cc:227] Handling call
0921 18:44:49.515045 (+ 136us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:44:49.515114 (+ 69us) raft_consensus.cc:1520] Early marking committed up to index 0
0921 18:44:49.515117 (+ 3us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:44:49.515215 (+ 98us) write_op.cc:270] Start()
0921 18:44:50.235366 (+720151us) write_op.cc:276] Timestamp: P: 1790016287972754 usec, L: 0
0921 18:44:50.235713 (+ 347us) write_op.cc:270] Start()
0921 18:44:51.131846 (+896133us) write_op.cc:276] Timestamp: P: 1790016288799558 usec, L: 0
0921 18:44:51.132298 (+ 452us) log.cc:842] Serialized 708 byte log entry
0921 18:44:51.132415 (+ 117us) raft_consensus.cc:1631] Marking committed up to 34
0921 18:44:51.132544 (+ 129us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:44:51.132573 (+ 29us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:44:51.132799 (+ 226us) raft_consensus.cc:1671] finished
0921 18:44:51.132801 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:44:51.132834 (+ 33us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:50.235709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:50.235795 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:50.235798 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:50.235800 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:50.235863 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:50.235872 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:50.235874 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:50.235908 (+ 34us) tablet.cc:699] Row locks acquired
0921 18:44:50.235909 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:51.132448 (+896539us) op_driver.cc:444] REPLICATION: finished
0921 18:44:51.132642 (+ 194us) write_op.cc:301] APPLY: starting
0921 18:44:51.132655 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:51.132678 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:51.132679 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:51.132728 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:51.132752 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:51.132769 (+ 17us) write_op.cc:312] APPLY: finished
Related trace 'op':
0921 18:44:51.132156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:51.132349 (+ 193us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:51.132353 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:51.132354 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:51.132415 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:51.132423 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:51.132425 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:51.132458 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:44:51.132459 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":157,"child_traces":[["op",{"apply.queue_time_us":154,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":223,"prepare.run_cpu_time_us":268,"prepare.run_wall_time_us":268,"replication_time_us":1617219,"thread_start_us":165,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":172,"prepare.run_cpu_time_us":282,"prepare.run_wall_time_us":370,"thread_start_us":76,"threads_started":1}]]}
I20260921 18:44:51.688876 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:51.689042 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:51.689569 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:51.689893 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:51.690107 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:51.690692 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:51.691061 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:51.692028 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.849s user 0.001s sys 0.000s
W20260921 18:44:51.692513 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.850s user 0.000s sys 0.000s
I20260921 18:44:52.023701 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:52.023821 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:52.024291 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:52.024641 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:52.024844 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:52.025175 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:52.025338 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:52.254302 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:52.254449 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:52.254974 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:52.255368 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:52.255373 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:52.255803 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:52.381758 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:52.381891 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:52.382462 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:52.382786 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:52.382840 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:52.383265 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:52.383461 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:52.383929 5411 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=37, attempt_no=0}) took 1295 ms. Trace:
I20260921 18:44:52.383996 5411 rpcz_store.cc:276] 0921 18:44:51.088544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:51.088597 (+ 53us) service_pool.cc:227] Handling call
0921 18:44:52.383917 (+1295320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:51.379257 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:51.379369 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:51.379373 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:51.379375 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:51.379453 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:51.379462 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:51.379464 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:51.379505 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:51.379506 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:51.379553 (+ 47us) write_op.cc:270] Start()
0921 18:44:52.082692 (+703139us) write_op.cc:276] Timestamp: P: 1790016291379500 usec, L: 0
0921 18:44:52.082696 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:52.082914 (+ 218us) log.cc:842] Serialized 354 byte log entry
0921 18:44:52.382662 (+299748us) op_driver.cc:444] REPLICATION: finished
0921 18:44:52.382928 (+ 266us) write_op.cc:301] APPLY: starting
0921 18:44:52.382942 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:52.382971 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:52.382982 (+ 11us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:52.383034 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:52.383060 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:52.383063 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:44:52.383146 (+ 83us) log.cc:842] Serialized 22 byte log entry
0921 18:44:52.383838 (+ 692us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:52.383850 (+ 12us) write_op.cc:454] Released schema lock
0921 18:44:52.383855 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":219,"num_ops":1,"prepare.queue_time_us":290421,"prepare.run_cpu_time_us":638,"prepare.run_wall_time_us":703705,"replication_time_us":299946,"thread_start_us":80,"threads_started":1}]]}
W20260921 18:44:52.384528 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.542s user 0.001s sys 0.000s
W20260921 18:44:52.385211 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.542s user 0.000s sys 0.000s
I20260921 18:44:52.833719 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:53.197937 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:53.198091 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:53.198544 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:53.198889 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:53.199024 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:53.199540 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:53.199688 5416 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=38, attempt_no=0}) took 1507 ms. Trace:
I20260921 18:44:53.199779 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:53.199776 5416 rpcz_store.cc:276] 0921 18:44:51.692555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:51.692616 (+ 61us) service_pool.cc:227] Handling call
0921 18:44:53.199676 (+1507060us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:52.082985 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:52.083103 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:52.083107 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:52.083110 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:52.083181 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:52.083190 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:52.083193 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:52.083234 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:52.083236 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:52.083274 (+ 38us) write_op.cc:270] Start()
0921 18:44:52.567406 (+484132us) write_op.cc:276] Timestamp: P: 1790016292083214 usec, L: 0
0921 18:44:52.567409 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:52.567612 (+ 203us) log.cc:842] Serialized 354 byte log entry
0921 18:44:53.199043 (+631431us) op_driver.cc:444] REPLICATION: finished
0921 18:44:53.199275 (+ 232us) write_op.cc:301] APPLY: starting
0921 18:44:53.199292 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:53.199320 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:53.199335 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:53.199400 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:53.199436 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:53.199441 (+ 5us) write_op.cc:312] APPLY: finished
0921 18:44:53.199513 (+ 72us) log.cc:842] Serialized 22 byte log entry
0921 18:44:53.199592 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:53.199604 (+ 12us) write_op.cc:454] Released schema lock
0921 18:44:53.199610 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":181,"num_ops":1,"prepare.queue_time_us":390055,"prepare.run_cpu_time_us":623,"prepare.run_wall_time_us":484693,"replication_time_us":631614,"thread_start_us":89,"threads_started":1}]]}
W20260921 18:44:53.200911 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.665s user 0.000s sys 0.000s
W20260921 18:44:53.201300 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.666s user 0.001s sys 0.000s
I20260921 18:44:53.276868 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:53.277007 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:53.277570 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:53.277828 4863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:53.277870 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:53.278323 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:53.278651 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:54.109200 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:54.109330 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:54.109859 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:54.110220 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:54.110230 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:54.110682 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:54.110909 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:54.111666 5422 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=39, attempt_no=0}) took 1726 ms. Trace:
I20260921 18:44:54.111734 5422 rpcz_store.cc:276] 0921 18:44:52.385642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:52.385694 (+ 52us) service_pool.cc:227] Handling call
0921 18:44:54.111650 (+1725956us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:52.567710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:52.567799 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:52.567805 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:44:52.567808 (+ 3us) tablet.cc:660] Decoding operations
0921 18:44:52.567891 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:52.567903 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:52.567907 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:52.567946 (+ 39us) tablet.cc:699] Row locks acquired
0921 18:44:52.567947 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:52.567983 (+ 36us) write_op.cc:270] Start()
0921 18:44:53.436120 (+868137us) write_op.cc:276] Timestamp: P: 1790016292567923 usec, L: 0
0921 18:44:53.436124 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:53.436364 (+ 240us) log.cc:842] Serialized 354 byte log entry
0921 18:44:54.110576 (+674212us) op_driver.cc:444] REPLICATION: finished
0921 18:44:54.110796 (+ 220us) write_op.cc:301] APPLY: starting
0921 18:44:54.110812 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:54.110841 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:54.110853 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:54.110907 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:54.110934 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:54.110938 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:54.110997 (+ 59us) log.cc:842] Serialized 22 byte log entry
0921 18:44:54.111096 (+ 99us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:54.111106 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:54.111111 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":166,"num_ops":1,"prepare.queue_time_us":181700,"prepare.run_cpu_time_us":671,"prepare.run_wall_time_us":868726,"raft.queue_time_us":140,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":674431,"thread_start_us":90,"threads_started":1}]]}
W20260921 18:44:54.112550 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.761s user 0.000s sys 0.000s
W20260921 18:44:54.113118 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.762s user 0.000s sys 0.000s
I20260921 18:44:54.161713 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:54.161835 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:54.162338 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:54.162744 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:54.162760 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:54.163302 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:54.284343 5329 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:54.284476 5329 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:54.284983 5329 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:54.285377 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:54.285408 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:54.285951 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:54.286130 5329 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:54.286505 5422 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=40, attempt_no=0}) took 1085 ms. Trace:
I20260921 18:44:54.286564 5422 rpcz_store.cc:276] 0921 18:44:53.201423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:53.201492 (+ 69us) service_pool.cc:227] Handling call
0921 18:44:54.286497 (+1085005us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:53.436470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:53.436577 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:53.436581 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:53.436582 (+ 1us) tablet.cc:660] Decoding operations
0921 18:44:53.436679 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:53.436689 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:53.436695 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:53.436739 (+ 44us) tablet.cc:699] Row locks acquired
0921 18:44:53.436740 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:53.436779 (+ 39us) write_op.cc:270] Start()
0921 18:44:53.733912 (+297133us) write_op.cc:276] Timestamp: P: 1790016293436731 usec, L: 0
0921 18:44:53.733915 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:53.734100 (+ 185us) log.cc:842] Serialized 354 byte log entry
0921 18:44:54.285911 (+551811us) op_driver.cc:444] REPLICATION: finished
0921 18:44:54.286022 (+ 111us) write_op.cc:301] APPLY: starting
0921 18:44:54.286038 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:54.286060 (+ 22us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:54.286076 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:54.286141 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:54.286177 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:54.286197 (+ 20us) write_op.cc:312] APPLY: finished
0921 18:44:54.286257 (+ 60us) log.cc:842] Serialized 22 byte log entry
0921 18:44:54.286424 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:54.286435 (+ 11us) write_op.cc:454] Released schema lock
0921 18:44:54.286447 (+ 12us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"num_ops":1,"prepare.queue_time_us":234665,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":297680,"replication_time_us":551361}]]}
I20260921 18:44:54.806031 5426 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:54.806173 5426 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:54.807044 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:54.807217 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:54.806762 5426 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:54.807649 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:54.807775 5426 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:54.809348 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.546s user 0.000s sys 0.000s
W20260921 18:44:54.809373 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.546s user 0.001s sys 0.000s
I20260921 18:44:54.825865 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:55.112890 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:55.113031 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:55.113507 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:55.113874 4863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:55.113920 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:55.114326 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:55.114485 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:55.526150 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:55.526288 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:55.526859 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:55.527191 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:55.527240 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:55.527577 5434 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=42, attempt_no=0}) took 1239 ms. Trace:
I20260921 18:44:55.527669 5434 rpcz_store.cc:276] 0921 18:44:54.287663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:54.287717 (+ 54us) service_pool.cc:227] Handling call
0921 18:44:55.527566 (+1239849us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:54.792920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:54.793003 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:54.793006 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:44:54.793008 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:54.793097 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:54.793106 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:54.793109 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:54.793148 (+ 39us) tablet.cc:699] Row locks acquired
0921 18:44:54.793150 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:44:54.793184 (+ 34us) write_op.cc:270] Start()
0921 18:44:55.482323 (+689139us) write_op.cc:276] Timestamp: P: 1790016294793148 usec, L: 0
0921 18:44:55.482327 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:44:55.482535 (+ 208us) log.cc:842] Serialized 354 byte log entry
0921 18:44:55.526965 (+ 44430us) op_driver.cc:444] REPLICATION: finished
0921 18:44:55.527231 (+ 266us) write_op.cc:301] APPLY: starting
0921 18:44:55.527246 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:55.527273 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:55.527289 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:55.527339 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:55.527363 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:55.527366 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:44:55.527426 (+ 60us) log.cc:842] Serialized 22 byte log entry
0921 18:44:55.527495 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:55.527504 (+ 9us) write_op.cc:454] Released schema lock
0921 18:44:55.527509 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":213,"num_ops":1,"prepare.queue_time_us":504879,"prepare.run_cpu_time_us":606,"prepare.run_wall_time_us":689675,"replication_time_us":44614,"thread_start_us":97,"threads_started":1}]]}
I20260921 18:44:55.527958 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:55.528268 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:55.529011 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.569s user 0.001s sys 0.000s
W20260921 18:44:55.529086 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.569s user 0.001s sys 0.000s
I20260921 18:44:55.927243 5433 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:55.927371 5433 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:55.927973 5433 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:55.928311 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:55.928380 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:55.928987 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:56.650211 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:56.650385 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:56.651036 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:56.651381 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:56.651387 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:56.651819 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:56.652017 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:56.652925 5442 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=43, attempt_no=0}) took 1843 ms. Trace:
W20260921 18:44:56.653285 5220 scanner-internal.cc:458] Time spent opening tablet: real 0.974s user 0.001s sys 0.000s
W20260921 18:44:56.653328 5219 scanner-internal.cc:458] Time spent opening tablet: real 0.974s user 0.001s sys 0.000s
I20260921 18:44:56.652997 5442 rpcz_store.cc:276] 0921 18:44:54.809291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:54.809366 (+ 75us) service_pool.cc:227] Handling call
0921 18:44:56.652912 (+1843546us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:55.482619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:55.482750 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:55.482755 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:44:55.482757 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:55.482838 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:55.482858 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:55.482861 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:55.482902 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:44:55.482906 (+ 4us) write_op.cc:260] PREPARE: finished
0921 18:44:55.482953 (+ 47us) write_op.cc:270] Start()
0921 18:44:55.846078 (+363125us) write_op.cc:276] Timestamp: P: 1790016295482908 usec, L: 0
0921 18:44:55.846081 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:55.846355 (+ 274us) log.cc:842] Serialized 354 byte log entry
0921 18:44:56.651469 (+805114us) op_driver.cc:444] REPLICATION: finished
0921 18:44:56.651763 (+ 294us) write_op.cc:301] APPLY: starting
0921 18:44:56.651776 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:56.651802 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:56.651816 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:56.651870 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:56.651897 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:56.651901 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:56.651962 (+ 61us) log.cc:842] Serialized 22 byte log entry
0921 18:44:56.652820 (+ 858us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:56.652830 (+ 10us) write_op.cc:454] Released schema lock
0921 18:44:56.652835 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":239,"num_ops":1,"prepare.queue_time_us":672913,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":363789,"replication_time_us":805368,"thread_start_us":62,"threads_started":1}]]}
I20260921 18:44:56.720558 5433 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:57.073462 5370 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:57.073640 5370 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.074184 5370 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:57.074564 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:57.074647 4864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:57.075114 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:57.075387 5442 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59814 (ReqId={client: a00a04137e4943f49bd16c4ff56c413b, seq_no=44, attempt_no=0}) took 1545 ms. Trace:
I20260921 18:44:57.075611 5370 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:57.075457 5442 rpcz_store.cc:276] 0921 18:44:55.529425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:55.529504 (+ 79us) service_pool.cc:227] Handling call
0921 18:44:57.075376 (+1545872us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:55.846452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 596bd65c0a4d44f5a1cc6390db4f0757
0921 18:44:55.846559 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:55.846563 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:44:55.846572 (+ 9us) tablet.cc:660] Decoding operations
0921 18:44:55.846643 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:55.846651 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:55.846654 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:55.846697 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:44:55.846698 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:55.846725 (+ 27us) write_op.cc:270] Start()
0921 18:44:56.766832 (+920107us) write_op.cc:276] Timestamp: P: 1790016295846702 usec, L: 0
0921 18:44:56.766835 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:44:56.766963 (+ 128us) log.cc:842] Serialized 354 byte log entry
0921 18:44:57.074414 (+307451us) op_driver.cc:444] REPLICATION: finished
0921 18:44:57.075011 (+ 597us) write_op.cc:301] APPLY: starting
0921 18:44:57.075028 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:44:57.075058 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:44:57.075071 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:44:57.075123 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:44:57.075149 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:44:57.075153 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:44:57.075210 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:44:57.075270 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:44:57.075302 (+ 32us) write_op.cc:454] Released schema lock
0921 18:44:57.075306 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":546,"num_ops":1,"prepare.queue_time_us":316703,"prepare.run_cpu_time_us":562,"prepare.run_wall_time_us":920618,"raft.queue_time_us":88,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":346,"replication_time_us":307558,"spinlock_wait_cycles":27520}]]}
I20260921 18:44:57.134935 4975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:44:57.135455 5106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:44:57.677464 5433 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:57.677623 5433 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.678226 5433 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:57.678507 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:57.678786 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:57.679244 4799 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b59277406044fc984fffda4ef77a363; no voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca
I20260921 18:44:57.679425 5433 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:57.680263 4844 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
3d3d4837f42341be88af4e62b6864371 | 127.3.36.190:42281 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:35593 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2b59277406044fc984fffda4ef77a363 | 127.3.36.129:36309 | HEALTHY | <none> | 0 | 0
9341d0fb8bc44c10ab9bad3c1818d831 | 127.3.36.130:38349 | HEALTHY | <none> | 1 | 0
beb9da40af4040bfba9e185697c750ca | 127.3.36.131:34241 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:36309
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:38349
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:34241
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:36309
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:38349
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:34241
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:35593 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260921 18:44:57.703878 3218 log_verifier.cc:126] Checking tablet 596bd65c0a4d44f5a1cc6390db4f0757
I20260921 18:44:57.739197 5433 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:57.739331 5433 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.739847 5433 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:57.740221 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:57.740345 5126 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b59277406044fc984fffda4ef77a363 in term 1.
I20260921 18:44:57.740713 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b59277406044fc984fffda4ef77a363, beb9da40af4040bfba9e185697c750ca; no voters:
I20260921 18:44:57.740737 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:57.740942 5433 raft_consensus.cc:2781] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260921 18:44:57.741014 5433 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9341d0fb8bc44c10ab9bad3c1818d831)
I20260921 18:44:57.741068 5433 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:44:57.742245 5433 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.742566 5433 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 election: Requested vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241)
I20260921 18:44:57.743119 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca"
I20260921 18:44:57.743218 5126 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:44:57.743192 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "2b59277406044fc984fffda4ef77a363" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831"
I20260921 18:44:57.743731 3218 log_verifier.cc:177] Verified matching terms for 46 ops in tablet 596bd65c0a4d44f5a1cc6390db4f0757
I20260921 18:44:57.744204 5126 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b59277406044fc984fffda4ef77a363 in term 2.
I20260921 18:44:57.744490 4800 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b59277406044fc984fffda4ef77a363, beb9da40af4040bfba9e185697c750ca; no voters:
I20260921 18:44:57.744748 5433 raft_consensus.cc:2781] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 2 FOLLOWER]: Leader election won for term 2
I20260921 18:44:57.744911 5433 raft_consensus.cc:686] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 2 LEADER]: Becoming Leader. State: Replica: 2b59277406044fc984fffda4ef77a363, State: Running, Role: LEADER
I20260921 18:44:57.745240 5433 consensus_queue.cc:237] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 46, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.745675 5477 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:57.745903 5477 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.746285 5477 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:57.746309 5126 raft_consensus.cc:1226] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Rejecting Update request from peer 9341d0fb8bc44c10ab9bad3c1818d831 for earlier term 1. Current term is 2. Ops: []
I20260921 18:44:57.746487 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 3 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:57.746634 4863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 3 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "2b59277406044fc984fffda4ef77a363" is_pre_election: true
I20260921 18:44:57.746901 5440 consensus_queue.cc:1046] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 }, Status: INVALID_TERM, Last received: 1.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260921 18:44:57.747023 5062 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: beb9da40af4040bfba9e185697c750ca; no voters: 2b59277406044fc984fffda4ef77a363, 9341d0fb8bc44c10ab9bad3c1818d831
I20260921 18:44:57.747166 5477 raft_consensus.cc:2726] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:44:57.747249 5474 raft_consensus.cc:3032] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:44:57.747308 5474 raft_consensus.cc:729] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9341d0fb8bc44c10ab9bad3c1818d831, State: Running, Role: LEADER
I20260921 18:44:57.747462 5474 consensus_queue.cc:260] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 46, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.747727 5474 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:44:57.748504 4863 raft_consensus.cc:1226] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [term 2 LEADER]: Rejecting Update request from peer 9341d0fb8bc44c10ab9bad3c1818d831 for earlier term 1. Current term is 2. Ops: []
I20260921 18:44:57.749394 4722 catalog_manager.cc:5495] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 reported cstate change: term changed from 1 to 2, leader changed from 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130) to 2b59277406044fc984fffda4ef77a363 (127.3.36.129). New cstate: current_term: 2 leader_uuid: "2b59277406044fc984fffda4ef77a363" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } health_report { overall_health: HEALTHY } } }
I20260921 18:44:57.750517 4995 raft_consensus.cc:1261] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Refusing update from remote peer 2b59277406044fc984fffda4ef77a363: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 2 index: 47. (index mismatch)
I20260921 18:44:57.751027 5126 raft_consensus.cc:1261] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Refusing update from remote peer 2b59277406044fc984fffda4ef77a363: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 2 index: 47. (index mismatch)
I20260921 18:44:57.751060 5483 consensus_queue.cc:1035] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260921 18:44:57.752560 5483 consensus_queue.cc:1035] T 596bd65c0a4d44f5a1cc6390db4f0757 P 2b59277406044fc984fffda4ef77a363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20260921 18:44:57.769979 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 4782
I20260921 18:44:57.775131 5474 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b59277406044fc984fffda4ef77a363)
I20260921 18:44:57.775255 5474 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.775738 5474 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:57.775895 5492 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b59277406044fc984fffda4ef77a363)
I20260921 18:44:57.775983 5492 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.776228 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca" is_pre_election: true
I20260921 18:44:57.776389 5126 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in term 2.
I20260921 18:44:57.776427 5492 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
I20260921 18:44:57.776659 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "beb9da40af4040bfba9e185697c750ca" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" is_pre_election: true
I20260921 18:44:57.776710 4931 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca; no voters:
I20260921 18:44:57.776793 4995 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beb9da40af4040bfba9e185697c750ca in term 2.
I20260921 18:44:57.776875 5474 raft_consensus.cc:2781] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260921 18:44:57.776916 5474 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2b59277406044fc984fffda4ef77a363)
I20260921 18:44:57.776968 5474 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 2 FOLLOWER]: Advancing to term 3
I20260921 18:44:57.777878 5474 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.778167 5474 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 3 election: Requested vote from peers beb9da40af4040bfba9e185697c750ca (127.3.36.131:34241), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
W20260921 18:44:57.778367 5062 connection.cc:535] client connection to 127.3.36.129:36309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:44:57.778460 5061 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca; no voters:
I20260921 18:44:57.778594 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596bd65c0a4d44f5a1cc6390db4f0757" candidate_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "beb9da40af4040bfba9e185697c750ca"
I20260921 18:44:57.778676 5492 raft_consensus.cc:2781] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260921 18:44:57.778793 5126 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 2 FOLLOWER]: Advancing to term 3
I20260921 18:44:57.779783 5126 raft_consensus.cc:2454] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9341d0fb8bc44c10ab9bad3c1818d831 in term 3.
W20260921 18:44:57.778493 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:44:57.779888 5492 raft_consensus.cc:482] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:44:57.779970 5492 raft_consensus.cc:3037] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 3 FOLLOWER]: Advancing to term 4
I20260921 18:44:57.780113 4931 leader_election.cc:302] T 596bd65c0a4d44f5a1cc6390db4f0757 P 9341d0fb8bc44c10ab9bad3c1818d831 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9341d0fb8bc44c10ab9bad3c1818d831, beb9da40af4040bfba9e185697c750ca; no voters:
I20260921 18:44:57.780789 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 4913
W20260921 18:44:57.780817 4931 connection.cc:535] client connection to 127.3.36.129:36309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:44:57.783910 5492 raft_consensus.cc:504] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9341d0fb8bc44c10ab9bad3c1818d831" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38349 } } peers { permanent_uuid: "beb9da40af4040bfba9e185697c750ca" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34241 } } peers { permanent_uuid: "2b59277406044fc984fffda4ef77a363" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 36309 } }
I20260921 18:44:57.784353 5492 leader_election.cc:288] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 4 election: Requested vote from peers 9341d0fb8bc44c10ab9bad3c1818d831 (127.3.36.130:38349), 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309)
W20260921 18:44:57.784718 5062 leader_election.cc:334] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309): Network error: Client connection negotiation failed: client connection to 127.3.36.129:36309: connect: Connection refused (error 111)
W20260921 18:44:57.786303 5062 leader_election.cc:334] T 596bd65c0a4d44f5a1cc6390db4f0757 P beb9da40af4040bfba9e185697c750ca [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 2b59277406044fc984fffda4ef77a363 (127.3.36.129:36309): Network error: Client connection negotiation failed: client connection to 127.3.36.129:36309: connect: Connection refused (error 111)
W20260921 18:44:57.789724 5061 connection.cc:535] client connection to 127.3.36.130:38349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:44:57.789999 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 5044
I20260921 18:44:57.795755 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 4691
2026-09-21T18:44:57Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (32832 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20260921 18:44:57.819388 3218 test_util.cc:268] Using random seed: -1517101832
I20260921 18:44:57.820109 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:44:57.820144 3218 ts_itest-base.cc:113] --------------
I20260921 18:44:57.820156 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:44:57.820168 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:44:57.820178 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:44:57Z Disabled control of system clock
I20260921 18:44:57.828689 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:33165
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:33165 with env {}
W20260921 18:44:57.934520 5505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:57.934760 5505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:57.934818 5505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:57.938692 5505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:44:57.938759 5505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:57.938789 5505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:44:57.938839 5505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:44:57.943239 5505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:33165
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:57.944181 5505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:57.945392 5505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260921 18:44:57.953583 5505 server_base.cc:1034] running on GCE node
W20260921 18:44:57.953517 5510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:57.953540 5511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:57.953526 5513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:57.954262 5505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:57.955267 5505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:57.956444 5505 hybrid_clock.cc:648] HybridClock initialized: now 1790016297956433 us; error 32 us; skew 500 ppm
I20260921 18:44:57.958148 5505 webserver.cc:460] Webserver started at http://127.3.36.190:46605/ using document root <none> and password file <none>
I20260921 18:44:57.958706 5505 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:57.958794 5505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:57.959015 5505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:57.960783 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5b4c8580482948e19362fe2e4df6ba0c"
format_stamp: "Formatted at 2026-09-21 18:44:57 on dist-test-slave-l4st"
I20260921 18:44:57.961326 5505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5b4c8580482948e19362fe2e4df6ba0c"
format_stamp: "Formatted at 2026-09-21 18:44:57 on dist-test-slave-l4st"
I20260921 18:44:57.964773 5505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260921 18:44:57.966975 5519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:57.968079 5505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:44:57.968207 5505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5b4c8580482948e19362fe2e4df6ba0c"
format_stamp: "Formatted at 2026-09-21 18:44:57 on dist-test-slave-l4st"
I20260921 18:44:57.968307 5505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:57.984048 5505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:57.984627 5505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:57.984812 5505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:57.991166 5505 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:33165
I20260921 18:44:57.991173 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:33165 every 8 connection(s)
I20260921 18:44:57.992005 5505 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:44:57.993721 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 5505
I20260921 18:44:57.993842 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:44:57.995398 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:58.000305 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c: Bootstrap starting.
I20260921 18:44:58.002549 5572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:58.003333 5572 log.cc:824] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:58.004968 5572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c: No bootstrap required, opened a new log
I20260921 18:44:58.008216 5572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4c8580482948e19362fe2e4df6ba0c" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 33165 } }
I20260921 18:44:58.008409 5572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:58.008487 5572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4c8580482948e19362fe2e4df6ba0c, State: Initialized, Role: FOLLOWER
I20260921 18:44:58.008857 5572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4c8580482948e19362fe2e4df6ba0c" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 33165 } }
I20260921 18:44:58.008988 5572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:44:58.009073 5572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:44:58.009186 5572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:58.010114 5572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4c8580482948e19362fe2e4df6ba0c" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 33165 } }
I20260921 18:44:58.010433 5572 leader_election.cc:302] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4c8580482948e19362fe2e4df6ba0c; no voters:
I20260921 18:44:58.010731 5572 leader_election.cc:288] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:44:58.010794 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:44:58.011071 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [term 1 LEADER]: Becoming Leader. State: Replica: 5b4c8580482948e19362fe2e4df6ba0c, State: Running, Role: LEADER
I20260921 18:44:58.011348 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4c8580482948e19362fe2e4df6ba0c" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 33165 } }
I20260921 18:44:58.011833 5572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:44:58.012953 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4c8580482948e19362fe2e4df6ba0c. Latest consensus state: current_term: 1 leader_uuid: "5b4c8580482948e19362fe2e4df6ba0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4c8580482948e19362fe2e4df6ba0c" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 33165 } } }
I20260921 18:44:58.013068 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [sys.catalog]: This master's current role is: LEADER
I20260921 18:44:58.013422 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:44:58.014034 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4c8580482948e19362fe2e4df6ba0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4c8580482948e19362fe2e4df6ba0c" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 33165 } } }
I20260921 18:44:58.014165 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c [sys.catalog]: This master's current role is: LEADER
I20260921 18:44:58.016064 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:44:58.021392 5586 catalog_manager.cc:1347] Generated new cluster ID: 9eb91d38f1964dbbb7a8e6759fc9d351
I20260921 18:44:58.021456 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:44:58.041128 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:44:58.042011 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:44:58.046638 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b4c8580482948e19362fe2e4df6ba0c: Generated new TSK 0
I20260921 18:44:58.047310 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:44:58.058677 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:33165
--builtin_ntp_servers=127.3.36.148:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20260921 18:44:58.168502 5596 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260921 18:44:58.168740 5596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:58.168759 5596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:58.168828 5596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:58.172987 5596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:58.173072 5596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:44:58.177804 5596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:33165
--never_fsync=true
--heap_profile_path=/tmp/kudu.5596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:58.178791 5596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:58.180106 5596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:58.188879 5602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:58.188882 5601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:58.188916 5596 server_base.cc:1034] running on GCE node
W20260921 18:44:58.188903 5604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:58.189550 5596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:58.190086 5596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:58.191267 5596 hybrid_clock.cc:648] HybridClock initialized: now 1790016298191242 us; error 47 us; skew 500 ppm
I20260921 18:44:58.192883 5596 webserver.cc:460] Webserver started at http://127.3.36.129:35883/ using document root <none> and password file <none>
I20260921 18:44:58.193420 5596 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:58.193538 5596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:58.193758 5596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:58.195533 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "37a20c5ed1254fd09d786fe4099ed76a"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.196067 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "37a20c5ed1254fd09d786fe4099ed76a"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.199455 5596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260921 18:44:58.201704 5610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.202814 5596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:44:58.202943 5596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "37a20c5ed1254fd09d786fe4099ed76a"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.203045 5596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:58.211782 5596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:58.212352 5596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:58.212584 5596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:58.213095 5596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:58.214064 5596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:58.214133 5596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.214200 5596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:58.214252 5596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.222724 5596 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:38133
I20260921 18:44:58.222744 5723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:38133 every 8 connection(s)
I20260921 18:44:58.223537 5596 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:44:58.225107 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 5596
I20260921 18:44:58.225250 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:44:58.227514 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:33165
--builtin_ntp_servers=127.3.36.148:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260921 18:44:58.234318 5724 heartbeater.cc:347] Connected to a master server at 127.3.36.190:33165
I20260921 18:44:58.234573 5724 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:58.235059 5724 heartbeater.cc:511] Master 127.3.36.190:33165 requested a full tablet report, sending...
I20260921 18:44:58.236485 5536 ts_manager.cc:194] Registered new tserver with Master: 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133)
I20260921 18:44:58.237672 5536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:39825
W20260921 18:44:58.335044 5727 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260921 18:44:58.335276 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:58.335296 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:58.335368 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:58.339293 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:58.339373 5727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:44:58.343883 5727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:33165
--never_fsync=true
--heap_profile_path=/tmp/kudu.5727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:58.344853 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:58.346225 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:58.355057 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:58.355067 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:58.355122 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:58.355234 5727 server_base.cc:1034] running on GCE node
I20260921 18:44:58.355755 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:58.356318 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:58.357509 5727 hybrid_clock.cc:648] HybridClock initialized: now 1790016298357455 us; error 27 us; skew 500 ppm
I20260921 18:44:58.359297 5727 webserver.cc:460] Webserver started at http://127.3.36.130:46011/ using document root <none> and password file <none>
I20260921 18:44:58.359831 5727 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:58.359890 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:58.360117 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:58.361948 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2f8374f0de944897bfe1594dd47a2725"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.362483 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2f8374f0de944897bfe1594dd47a2725"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.365902 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:44:58.368127 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.369251 5727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:44:58.369385 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2f8374f0de944897bfe1594dd47a2725"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.369529 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:58.395980 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:58.396644 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:58.396827 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:58.397358 5727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:58.398283 5727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:58.398353 5727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.398416 5727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:58.398468 5727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.406944 5727 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:38815
I20260921 18:44:58.406993 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:38815 every 8 connection(s)
I20260921 18:44:58.407806 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:44:58.413633 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 5727
I20260921 18:44:58.413761 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:44:58.415897 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:33165
--builtin_ntp_servers=127.3.36.148:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20260921 18:44:58.418699 5855 heartbeater.cc:347] Connected to a master server at 127.3.36.190:33165
I20260921 18:44:58.418919 5855 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:58.419378 5855 heartbeater.cc:511] Master 127.3.36.190:33165 requested a full tablet report, sending...
I20260921 18:44:58.420492 5536 ts_manager.cc:194] Registered new tserver with Master: 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:44:58.421353 5536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:48057
W20260921 18:44:58.524428 5858 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20260921 18:44:58.524677 5858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:44:58.524696 5858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:44:58.524729 5858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:44:58.528659 5858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:44:58.528746 5858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:44:58.533319 5858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:33165
--never_fsync=true
--heap_profile_path=/tmp/kudu.5858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:44:58.534296 5858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:44:58.535568 5858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:44:58.544510 5863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:58.544524 5864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:44:58.544581 5866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:44:58.544852 5858 server_base.cc:1034] running on GCE node
I20260921 18:44:58.545167 5858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:44:58.545792 5858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:44:58.546947 5858 hybrid_clock.cc:648] HybridClock initialized: now 1790016298546931 us; error 26 us; skew 500 ppm
I20260921 18:44:58.548689 5858 webserver.cc:460] Webserver started at http://127.3.36.131:44801/ using document root <none> and password file <none>
I20260921 18:44:58.549222 5858 fs_manager.cc:359] Metadata directory not provided
I20260921 18:44:58.549281 5858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:44:58.549583 5858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:44:58.551386 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b9819191de3942c1aa42bbfdc863de79"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.551935 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b9819191de3942c1aa42bbfdc863de79"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.555397 5858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:44:58.557715 5872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.558826 5858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:44:58.558960 5858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b9819191de3942c1aa42bbfdc863de79"
format_stamp: "Formatted at 2026-09-21 18:44:58 on dist-test-slave-l4st"
I20260921 18:44:58.559067 5858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:44:58.577121 5858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:44:58.577836 5858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:44:58.578024 5858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:44:58.578564 5858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:44:58.579452 5858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:44:58.579528 5858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.579600 5858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:44:58.579644 5858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:44:58.588138 5858 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:41413
I20260921 18:44:58.588166 5985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:41413 every 8 connection(s)
I20260921 18:44:58.588977 5858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:44:58.592090 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 5858
I20260921 18:44:58.592232 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:44:58.598887 5986 heartbeater.cc:347] Connected to a master server at 127.3.36.190:33165
I20260921 18:44:58.599113 5986 heartbeater.cc:464] Registering TS with master...
I20260921 18:44:58.599570 5986 heartbeater.cc:511] Master 127.3.36.190:33165 requested a full tablet report, sending...
I20260921 18:44:58.600479 5536 ts_manager.cc:194] Registered new tserver with Master: b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:58.601027 5536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:44839
I20260921 18:44:58.606287 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:44:58.607709 3218 test_util.cc:268] Using random seed: -1516313511
I20260921 18:44:58.617776 5536 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53502:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260921 18:44:58.618851 5536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:44:58.635067 5658 tablet_service.cc:1467] Processing CreateTablet for tablet f094ab8ff7d84a9f96e61813d4559ac2 (DEFAULT_TABLE table=test-workload [id=d44aff99241b41e280681bc2570e93d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:58.635288 5920 tablet_service.cc:1467] Processing CreateTablet for tablet f094ab8ff7d84a9f96e61813d4559ac2 (DEFAULT_TABLE table=test-workload [id=d44aff99241b41e280681bc2570e93d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:58.636122 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f094ab8ff7d84a9f96e61813d4559ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:58.636138 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f094ab8ff7d84a9f96e61813d4559ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:58.636227 5789 tablet_service.cc:1467] Processing CreateTablet for tablet f094ab8ff7d84a9f96e61813d4559ac2 (DEFAULT_TABLE table=test-workload [id=d44aff99241b41e280681bc2570e93d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:44:58.637034 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f094ab8ff7d84a9f96e61813d4559ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:44:58.642146 6003 tablet_bootstrap.cc:492] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79: Bootstrap starting.
I20260921 18:44:58.642900 6005 tablet_bootstrap.cc:492] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725: Bootstrap starting.
I20260921 18:44:58.643244 6004 tablet_bootstrap.cc:492] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: Bootstrap starting.
I20260921 18:44:58.644491 6003 tablet_bootstrap.cc:654] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:58.644863 6004 tablet_bootstrap.cc:654] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:58.644909 6005 tablet_bootstrap.cc:654] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725: Neither blocks nor log segments found. Creating new log.
I20260921 18:44:58.645323 6003 log.cc:824] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:58.645570 6004 log.cc:824] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:58.645671 6005 log.cc:824] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725: Log is configured to *not* fsync() on all Append() calls
I20260921 18:44:58.647091 6003 tablet_bootstrap.cc:492] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79: No bootstrap required, opened a new log
I20260921 18:44:58.647269 6003 ts_tablet_manager.cc:1397] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:44:58.647359 6004 tablet_bootstrap.cc:492] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: No bootstrap required, opened a new log
I20260921 18:44:58.647531 6004 ts_tablet_manager.cc:1397] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260921 18:44:58.650547 6003 raft_consensus.cc:348] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.650738 6003 raft_consensus.cc:374] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:58.650790 6003 raft_consensus.cc:729] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9819191de3942c1aa42bbfdc863de79, State: Initialized, Role: FOLLOWER
I20260921 18:44:58.650676 6004 raft_consensus.cc:348] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.650955 6004 raft_consensus.cc:374] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:58.651023 6004 raft_consensus.cc:729] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a20c5ed1254fd09d786fe4099ed76a, State: Initialized, Role: FOLLOWER
I20260921 18:44:58.651326 6003 consensus_queue.cc:260] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.651494 6004 consensus_queue.cc:260] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.651842 6010 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:58.651954 6010 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.652217 6005 tablet_bootstrap.cc:492] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725: No bootstrap required, opened a new log
I20260921 18:44:58.652254 6003 ts_tablet_manager.cc:1428] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:44:58.652355 6004 ts_tablet_manager.cc:1428] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:44:58.652376 6005 ts_tablet_manager.cc:1397] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260921 18:44:58.652571 5986 heartbeater.cc:503] Master 127.3.36.190:33165 was elected leader, sending a full tablet report...
I20260921 18:44:58.652642 5724 heartbeater.cc:503] Master 127.3.36.190:33165 was elected leader, sending a full tablet report...
I20260921 18:44:58.652968 6010 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:58.653795 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:58.653926 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.654953 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
W20260921 18:44:58.660465 5612 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.36.130:38815 timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260921 18:44:58.660576 6005 raft_consensus.cc:348] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.660833 6005 raft_consensus.cc:374] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:44:58.660882 6005 raft_consensus.cc:729] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8374f0de944897bfe1594dd47a2725, State: Initialized, Role: FOLLOWER
I20260921 18:44:58.661345 6005 consensus_queue.cc:260] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.661736 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:58.661844 6017 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:44:58.661895 5678 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9819191de3942c1aa42bbfdc863de79 in term 0.
I20260921 18:44:58.661972 6017 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.662129 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "37a20c5ed1254fd09d786fe4099ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:44:58.662284 5809 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a20c5ed1254fd09d786fe4099ed76a in term 0.
I20260921 18:44:58.662319 6005 ts_tablet_manager.cc:1428] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
W20260921 18:44:58.662609 5876 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.36.129:38133 timed out after 0.007s (SENT)
I20260921 18:44:58.662657 5855 heartbeater.cc:503] Master 127.3.36.190:33165 was elected leader, sending a full tablet report...
I20260921 18:44:58.662748 5612 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a; no voters:
I20260921 18:44:58.663054 6012 raft_consensus.cc:2781] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:44:58.663184 6012 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:44:58.663280 6012 raft_consensus.cc:3037] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:58.663403 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "37a20c5ed1254fd09d786fe4099ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:44:58.663550 6017 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:58.663594 5940 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a20c5ed1254fd09d786fe4099ed76a in term 0.
I20260921 18:44:58.665397 6012 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.665777 6012 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [CANDIDATE]: Term 1 election: Requested vote from peers 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:58.666116 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "37a20c5ed1254fd09d786fe4099ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79"
I20260921 18:44:58.666220 5940 raft_consensus.cc:3037] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:58.666277 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "37a20c5ed1254fd09d786fe4099ed76a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725"
I20260921 18:44:58.666370 5809 raft_consensus.cc:3037] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:44:58.667380 5940 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a20c5ed1254fd09d786fe4099ed76a in term 1.
I20260921 18:44:58.667460 5809 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a20c5ed1254fd09d786fe4099ed76a in term 1.
I20260921 18:44:58.667752 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:58.667970 5678 raft_consensus.cc:2379] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9819191de3942c1aa42bbfdc863de79 in current term 1: Already voted for candidate 37a20c5ed1254fd09d786fe4099ed76a in this term.
I20260921 18:44:58.667795 5611 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79; no voters:
I20260921 18:44:58.668604 6012 raft_consensus.cc:2781] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:44:58.668764 6012 raft_consensus.cc:686] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [term 1 LEADER]: Becoming Leader. State: Replica: 37a20c5ed1254fd09d786fe4099ed76a, State: Running, Role: LEADER
I20260921 18:44:58.669044 6012 consensus_queue.cc:237] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.669839 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:58.670723 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:44:58.670897 5940 raft_consensus.cc:2379] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f8374f0de944897bfe1594dd47a2725 in current term 1: Already voted for candidate 37a20c5ed1254fd09d786fe4099ed76a in this term.
I20260921 18:44:58.671317 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:44:58.671463 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:44:58.671620 5809 raft_consensus.cc:2379] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9819191de3942c1aa42bbfdc863de79 in current term 1: Already voted for candidate 37a20c5ed1254fd09d786fe4099ed76a in this term.
I20260921 18:44:58.671877 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260921 18:44:58.671981 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:44:58.672405 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260921 18:44:58.673130 6010 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:44:58.673168 6010 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260921 18:44:58.673384 5536 catalog_manager.cc:5495] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129). New cstate: current_term: 1 leader_uuid: "37a20c5ed1254fd09d786fe4099ed76a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } } }
W20260921 18:44:58.724640 5725 tablet.cc:2369] T f094ab8ff7d84a9f96e61813d4559ac2 P 37a20c5ed1254fd09d786fe4099ed76a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:44:58.788163 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:44:58.788287 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.788645 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:44:58.789148 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:44:58.789402 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:58.789752 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:44:58.789983 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:44:58.837010 5638 tablet_service.cc:2908] Rejecting scan request for tablet f094ab8ff7d84a9f96e61813d4559ac2: Uninitialized: clean time has not yet been initialized
W20260921 18:44:58.837010 5637 tablet_service.cc:2908] Rejecting scan request for tablet f094ab8ff7d84a9f96e61813d4559ac2: Uninitialized: clean time has not yet been initialized
W20260921 18:44:58.842643 5808 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:44:58.842730 5808 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260921 18:44:58.843528 5940 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:44:58.843592 5940 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260921 18:44:58.844726 5769 tablet_service.cc:2970] error setting up scanner 84708bae8e864e6792d89927b1b065a1 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.149s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "e8275135b5b64a8891452d3a2922265a"
W20260921 18:44:58.844866 5768 tablet_service.cc:2970] error setting up scanner db6f9f6d0fad446f87fdb4adf2956f85 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.149s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
W20260921 18:44:58.869232 5768 tablet_service.cc:2970] error setting up scanner 8292fc4f1f204110ae49d1b748b6afd0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.174s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
W20260921 18:44:58.870826 5768 tablet_service.cc:2970] error setting up scanner fa1aa848265f4eebbe950e80840eda5d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.176s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "e8275135b5b64a8891452d3a2922265a"
W20260921 18:44:58.911116 5768 tablet_service.cc:2970] error setting up scanner ee2491af87ac49c4ad3099586f8c3ffb with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.216s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
W20260921 18:44:58.916352 5768 tablet_service.cc:2970] error setting up scanner 17f810f807944b3c8f46d20112708fda with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.221s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "e8275135b5b64a8891452d3a2922265a"
I20260921 18:44:58.979894 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:44:58.980023 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:58.980734 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:58.981312 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:58.981451 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:44:58.981984 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:44:58.982210 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:59.388906 6032 mvcc.cc:205] Tried to move back new op lower bound from 7331906759456137216 to 7331906759351869440. Current Snapshot: MvccSnapshot[applied={T|T < 7331906759456137216 or (T in {7331906759456137216})}]
I20260921 18:44:59.628448 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:44:59.628643 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:59.629153 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:59.629591 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:59.629627 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:44:59.630153 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:44:59.630323 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:59.681170 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:44:59.681282 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:59.681962 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:44:59.682649 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:59.682721 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:44:59.683164 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:44:59.683368 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:44:59.834625 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:44:59.834753 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:44:59.836050 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:44:59.836390 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:44:59.837113 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:44:59.837493 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:44:59.837713 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:00.054250 5768 tablet_service.cc:2970] error setting up scanner 23e592b5589b4b2b8cc1cb9ba7d63f8c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790016299053888 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790016298841505 usec, L: 0 Physical time difference: 0.212s: new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "e8275135b5b64a8891452d3a2922265a"
I20260921 18:45:00.054667 5768 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:57894 (request call id 6) took 1001 ms. Trace:
I20260921 18:45:00.054765 5768 rpcz_store.cc:276] 0921 18:44:59.053487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:59.053533 (+ 46us) service_pool.cc:227] Handling call
0921 18:44:59.053674 (+ 141us) tablet_service.cc:2780] Created scanner 23e592b5589b4b2b8cc1cb9ba7d63f8c for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is e8275135b5b64a8891452d3a2922265a
0921 18:44:59.053908 (+ 234us) tablet_service.cc:2920] Creating iterator
0921 18:44:59.053932 (+ 24us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:00.054218 (+1000286us) tablet_service.cc:2945] Iterator created
0921 18:45:00.054236 (+ 18us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790016299053888 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790016298841505 usec, L: 0 Physical time difference: 0.212s
0921 18:45:00.054652 (+ 416us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260921 18:45:00.064918 5769 tablet_service.cc:2970] error setting up scanner 50bbf36636034338a7c9e03ac44fd036 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790016299064649 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790016298841505 usec, L: 0 Physical time difference: 0.223s: new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:00.094184 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:00.094331 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:00.094805 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:00.095212 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:00.095288 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:00.095736 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
W20260921 18:45:00.291157 5769 tablet_service.cc:2970] error setting up scanner 85d58e2cec2d47ceb6ba7524415f79ec with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.661s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
W20260921 18:45:00.312980 5769 tablet_service.cc:2970] error setting up scanner ce51fcc9aecb417a838ff365bdeff1c8 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.683s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "e8275135b5b64a8891452d3a2922265a"
W20260921 18:45:00.613019 5769 tablet_service.cc:2970] error setting up scanner 397657e19e11459791ce8f16dad7206c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.983s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:00.815447 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:00.829530 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:00.829655 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:00.830197 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:00.830524 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:00.830528 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:00.830924 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:00.846197 6048 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=2, attempt_no=0}) took 1829 ms. Trace:
I20260921 18:45:00.846289 6048 rpcz_store.cc:276] 0921 18:44:59.016348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:59.016408 (+ 60us) service_pool.cc:227] Handling call
0921 18:45:00.846172 (+1829764us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:44:59.389129 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:44:59.389230 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:44:59.389237 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:44:59.389239 (+ 2us) tablet.cc:660] Decoding operations
0921 18:44:59.389312 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:44:59.389318 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:44:59.389320 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:44:59.389352 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:44:59.389353 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:44:59.389383 (+ 30us) write_op.cc:270] Start()
0921 18:45:00.002515 (+613132us) write_op.cc:276] Timestamp: P: 1790016299389349 usec, L: 0
0921 18:45:00.002520 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:00.002676 (+ 156us) log.cc:842] Serialized 354 byte log entry
0921 18:45:00.845556 (+842880us) op_driver.cc:444] REPLICATION: finished
0921 18:45:00.845775 (+ 219us) write_op.cc:301] APPLY: starting
0921 18:45:00.845788 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:00.845812 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:00.845826 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:00.845875 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:00.845896 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:00.845899 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:00.845954 (+ 55us) log.cc:842] Serialized 22 byte log entry
0921 18:45:00.846082 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:00.846093 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:00.846098 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":170,"num_ops":1,"prepare.queue_time_us":372255,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":613634,"raft.queue_time_us":355,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"replication_time_us":843010,"thread_start_us":76,"threads_started":1}]]}
I20260921 18:45:00.938844 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:00.984169 6048 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=3, attempt_no=0}) took 1306 ms. Trace:
I20260921 18:45:00.984304 6048 rpcz_store.cc:276] 0921 18:44:59.677971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:44:59.678027 (+ 56us) service_pool.cc:227] Handling call
0921 18:45:00.984152 (+1306125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:00.002794 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:00.002896 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:00.002900 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:00.002901 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:00.002962 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:00.002969 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:00.002971 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:00.003004 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:45:00.003005 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:00.003036 (+ 31us) write_op.cc:270] Start()
0921 18:45:00.590170 (+587134us) write_op.cc:276] Timestamp: P: 1790016300002991 usec, L: 0
0921 18:45:00.590174 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:00.590384 (+ 210us) log.cc:842] Serialized 354 byte log entry
0921 18:45:00.983736 (+393352us) op_driver.cc:444] REPLICATION: finished
0921 18:45:00.983832 (+ 96us) write_op.cc:301] APPLY: starting
0921 18:45:00.983850 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:00.983874 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:00.983890 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:00.983941 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:00.983964 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:00.983967 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:00.984035 (+ 68us) log.cc:842] Serialized 22 byte log entry
0921 18:45:00.984080 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:00.984090 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:00.984093 (+ 3us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":324448,"prepare.run_cpu_time_us":609,"prepare.run_wall_time_us":587674,"replication_time_us":393534}]]}
I20260921 18:45:01.033861 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:01.034010 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:01.034512 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:01.034858 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:01.034933 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:01.035367 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:01.132918 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:01.133064 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:01.133580 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:01.133828 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:01.134265 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:01.134676 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:01.156945 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:01.287926 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:01.288074 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:01.288600 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:01.288933 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:01.288949 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:01.289565 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:01.497156 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:01.574926 5768 tablet_service.cc:2970] error setting up scanner d34a759dfd5640d59c0a1f9a386215b6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.441s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:01.585964 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:01.591400 6030 scanner-internal.cc:458] Time spent opening tablet: real 2.757s user 0.003s sys 0.000s
I20260921 18:45:01.594784 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:01.594879 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:01.595206 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:01.595526 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:01.595645 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:01.596063 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:01.596238 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:01.906917 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:01.907059 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:01.907763 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:01.908102 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:01.908505 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:01.908903 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:01.909118 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:02.207559 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:02.207700 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:02.208195 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:02.208495 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:02.208756 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:02.209213 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:02.209378 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:02.618950 6009 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:02.619089 6009 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:02.619661 6009 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:02.620095 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:02.620110 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:02.620498 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
W20260921 18:45:03.369095 5769 tablet_service.cc:2970] error setting up scanner 348b97b4069049509e01e0e36773acf1 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.160s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:03.492291 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:03.492461 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:03.493036 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:03.493412 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:03.493431 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:03.493961 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:03.514158 6061 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=8, attempt_no=0}) took 1965 ms. Trace:
I20260921 18:45:03.514257 6061 rpcz_store.cc:276] 0921 18:45:01.548507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:01.548578 (+ 71us) service_pool.cc:227] Handling call
0921 18:45:03.514139 (+1965561us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:02.003262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:02.003364 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:02.003367 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:02.003368 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:02.003460 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:02.003472 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:02.003478 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:02.003522 (+ 44us) tablet.cc:699] Row locks acquired
0921 18:45:02.003523 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:02.003560 (+ 37us) write_op.cc:270] Start()
0921 18:45:02.559723 (+556163us) write_op.cc:276] Timestamp: P: 1790016302003509 usec, L: 0
0921 18:45:02.559727 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:02.559947 (+ 220us) log.cc:842] Serialized 354 byte log entry
0921 18:45:03.513548 (+953601us) op_driver.cc:444] REPLICATION: finished
0921 18:45:03.513754 (+ 206us) write_op.cc:301] APPLY: starting
0921 18:45:03.513785 (+ 31us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:03.513814 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:03.513838 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:03.513886 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:03.513909 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:03.513912 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:03.513965 (+ 53us) log.cc:842] Serialized 22 byte log entry
0921 18:45:03.514054 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:03.514065 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:03.514069 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"num_ops":1,"prepare.queue_time_us":454346,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":556743,"replication_time_us":953795,"thread_start_us":65,"threads_started":1}]]}
I20260921 18:45:03.515148 5638 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 13) took 1123 ms. Trace:
I20260921 18:45:03.515229 5638 rpcz_store.cc:276] 0921 18:45:02.392121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:02.392190 (+ 69us) service_pool.cc:227] Handling call
0921 18:45:02.392318 (+ 128us) tablet_service.cc:2780] Created scanner 538e14be547a4807af37c10f9e6dfa7d for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 8269ba3e3edf493299dd24fd1ef9af81
0921 18:45:02.392622 (+ 304us) tablet_service.cc:2920] Creating iterator
0921 18:45:02.392646 (+ 24us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:02.559833 (+167187us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:03.514092 (+954259us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1121435 microseconds
0921 18:45:03.514167 (+ 75us) tablet_service.cc:2945] Iterator created
0921 18:45:03.514647 (+ 480us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:03.514675 (+ 28us) tablet_service.cc:3010] has_more: true
0921 18:45:03.514707 (+ 32us) tablet_service.cc:3027] Continuing scan request
0921 18:45:03.514756 (+ 49us) tablet_service.cc:3091] Found scanner 538e14be547a4807af37c10f9e6dfa7d for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 8269ba3e3edf493299dd24fd1ef9af81
0921 18:45:03.515141 (+ 385us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:03.515648 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.124s user 0.001s sys 0.000s
I20260921 18:45:03.557531 6009 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:03.567903 6061 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=9, attempt_no=0}) took 1329 ms. Trace:
I20260921 18:45:03.567977 6061 rpcz_store.cc:276] 0921 18:45:02.238066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:02.238110 (+ 44us) service_pool.cc:227] Handling call
0921 18:45:03.567890 (+1329780us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:02.560044 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:02.560142 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:02.560146 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:02.560148 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:02.562367 (+ 2219us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:02.562381 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:02.562386 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:02.562435 (+ 49us) tablet.cc:699] Row locks acquired
0921 18:45:02.562436 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:02.562498 (+ 62us) write_op.cc:270] Start()
0921 18:45:02.586672 (+ 24174us) write_op.cc:276] Timestamp: P: 1790016302562457 usec, L: 0
0921 18:45:02.586676 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:02.586836 (+ 160us) log.cc:842] Serialized 354 byte log entry
0921 18:45:03.567447 (+980611us) op_driver.cc:444] REPLICATION: finished
0921 18:45:03.567548 (+ 101us) write_op.cc:301] APPLY: starting
0921 18:45:03.567562 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:03.567586 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:03.567599 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:03.567671 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:03.567699 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:03.567705 (+ 6us) write_op.cc:312] APPLY: finished
0921 18:45:03.567762 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:45:03.567814 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:03.567823 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:03.567827 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"num_ops":1,"prepare.queue_time_us":321631,"prepare.run_cpu_time_us":677,"prepare.run_wall_time_us":26871,"replication_time_us":980740}]]}
I20260921 18:45:04.115272 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:04.391887 6018 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:04.392042 6018 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:04.392522 6018 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:04.392931 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:04.392988 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:04.393579 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:04.393782 6018 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:04.442662 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.776s user 0.001s sys 0.000s
I20260921 18:45:04.530328 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:04.530458 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:04.530959 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:04.531229 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:04.531437 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:04.532016 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:04.532181 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:05.105432 6073 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:05.105616 6073 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:05.106122 6073 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:05.106528 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:05.106578 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:05.107125 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:05.135911 6076 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=11, attempt_no=0}) took 1566 ms. Trace:
I20260921 18:45:05.136013 6076 rpcz_store.cc:276] 0921 18:45:03.569249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:03.569306 (+ 57us) service_pool.cc:227] Handling call
0921 18:45:05.135895 (+1566589us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:04.192868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:04.192992 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:04.192998 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:45:04.193000 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:04.193075 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:04.193084 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:04.193086 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:04.193129 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:45:04.193130 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:04.193163 (+ 33us) write_op.cc:270] Start()
0921 18:45:04.902280 (+709117us) write_op.cc:276] Timestamp: P: 1790016304193109 usec, L: 0
0921 18:45:04.902283 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:04.902489 (+ 206us) log.cc:842] Serialized 354 byte log entry
0921 18:45:05.135375 (+232886us) op_driver.cc:444] REPLICATION: finished
0921 18:45:05.135594 (+ 219us) write_op.cc:301] APPLY: starting
0921 18:45:05.135608 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:05.135633 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:05.135645 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:05.135694 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:05.135715 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:05.135718 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:05.135772 (+ 54us) log.cc:842] Serialized 22 byte log entry
0921 18:45:05.135825 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:05.135834 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:05.135838 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":171,"num_ops":1,"prepare.queue_time_us":623182,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":709682,"raft.queue_time_us":133,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":191,"replication_time_us":233067,"spinlock_wait_cycles":45952,"thread_start_us":67,"threads_started":1}]]}
W20260921 18:45:05.137033 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.544s user 0.001s sys 0.000s
I20260921 18:45:05.854210 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:05.854360 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:05.854867 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:05.855269 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:05.855340 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:05.855810 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:05.977166 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:06.010891 6083 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=12, attempt_no=0}) took 1568 ms. Trace:
I20260921 18:45:06.011031 6083 rpcz_store.cc:276] 0921 18:45:04.442610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:04.442669 (+ 59us) service_pool.cc:227] Handling call
0921 18:45:06.010874 (+1568205us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:04.902611 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:04.902713 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:04.902720 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:45:04.902721 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:04.902794 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:04.902803 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:04.902806 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:04.902848 (+ 42us) tablet.cc:699] Row locks acquired
0921 18:45:04.902849 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:04.902888 (+ 39us) write_op.cc:270] Start()
0921 18:45:05.003024 (+100136us) write_op.cc:276] Timestamp: P: 1790016304902855 usec, L: 0
0921 18:45:05.003028 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:05.003219 (+ 191us) log.cc:842] Serialized 354 byte log entry
0921 18:45:06.010278 (+1007059us) op_driver.cc:444] REPLICATION: finished
0921 18:45:06.010482 (+ 204us) write_op.cc:301] APPLY: starting
0921 18:45:06.010515 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:06.010543 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:06.010566 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:06.010624 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:06.010672 (+ 48us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:06.010677 (+ 5us) write_op.cc:312] APPLY: finished
0921 18:45:06.010742 (+ 65us) log.cc:842] Serialized 22 byte log entry
0921 18:45:06.010789 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:06.010801 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:06.010806 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155,"num_ops":1,"prepare.queue_time_us":459606,"prepare.run_cpu_time_us":595,"prepare.run_wall_time_us":100662,"replication_time_us":1007227,"thread_start_us":76,"threads_started":1}]]}
W20260921 18:45:06.012313 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.725s user 0.001s sys 0.000s
I20260921 18:45:06.067768 6073 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:06.601897 6073 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:06.602039 6073 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:06.602509 6073 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:06.602996 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:06.603085 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:06.603511 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:06.686718 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:06.686847 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:06.687378 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:06.687790 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:06.687834 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:06.688282 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
W20260921 18:45:06.711815 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.549s user 0.001s sys 0.000s
I20260921 18:45:07.141355 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:07.157770 6092 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=15, attempt_no=0}) took 1145 ms. Trace:
I20260921 18:45:07.157850 6092 rpcz_store.cc:276] 0921 18:45:06.012682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:06.012762 (+ 80us) service_pool.cc:227] Handling call
0921 18:45:07.157754 (+1144992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:06.188411 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:06.188483 (+ 72us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:06.188486 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:06.188487 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:06.188544 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:06.188551 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:06.188552 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:06.188618 (+ 66us) tablet.cc:699] Row locks acquired
0921 18:45:06.188620 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:06.188655 (+ 35us) write_op.cc:270] Start()
0921 18:45:06.686798 (+498143us) write_op.cc:276] Timestamp: P: 1790016306188598 usec, L: 0
0921 18:45:06.686802 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:06.687021 (+ 219us) log.cc:842] Serialized 354 byte log entry
0921 18:45:07.157262 (+470241us) op_driver.cc:444] REPLICATION: finished
0921 18:45:07.157371 (+ 109us) write_op.cc:301] APPLY: starting
0921 18:45:07.157389 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:07.157425 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:07.157438 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:07.157538 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:07.157573 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:07.157577 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:07.157642 (+ 65us) log.cc:842] Serialized 22 byte log entry
0921 18:45:07.157687 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:07.157697 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:07.157701 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"num_ops":1,"prepare.queue_time_us":175409,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":498674,"replication_time_us":470436}]]}
I20260921 18:45:07.212080 6073 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:07.466898 5769 tablet_service.cc:2970] error setting up scanner ca7b22eaf59347068a3c794257f22ded with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.254s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:07.628155 6073 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:07.628322 6073 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:07.628823 6073 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:07.629160 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:07.629227 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:07.629623 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:07.649091 6073 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:07.679652 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.817s user 0.001s sys 0.000s
I20260921 18:45:08.114449 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:08.114574 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:08.115028 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:08.115311 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:08.115321 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:08.115782 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:08.115922 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:08.603360 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:08.603485 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:08.603995 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:08.604318 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:08.604408 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:08.604934 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:08.605083 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:08.632042 6100 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=17, attempt_no=0}) took 1472 ms. Trace:
I20260921 18:45:08.632143 6100 rpcz_store.cc:276] 0921 18:45:07.159154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:07.159190 (+ 36us) service_pool.cc:227] Handling call
0921 18:45:08.632024 (+1472834us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:07.518903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:07.518996 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:07.518999 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:07.519000 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:07.519099 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:07.519110 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:07.519112 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:07.519154 (+ 42us) tablet.cc:699] Row locks acquired
0921 18:45:07.519156 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:07.519190 (+ 34us) write_op.cc:270] Start()
0921 18:45:08.312335 (+793145us) write_op.cc:276] Timestamp: P: 1790016307519168 usec, L: 0
0921 18:45:08.312341 (+ 6us) op_driver.cc:331] REPLICATION: starting
0921 18:45:08.312602 (+ 261us) log.cc:842] Serialized 354 byte log entry
0921 18:45:08.631406 (+318804us) op_driver.cc:444] REPLICATION: finished
0921 18:45:08.631664 (+ 258us) write_op.cc:301] APPLY: starting
0921 18:45:08.631679 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:08.631704 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:08.631719 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:08.631772 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:08.631793 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:08.631796 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:08.631852 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:45:08.631932 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:08.631946 (+ 14us) write_op.cc:454] Released schema lock
0921 18:45:08.631951 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":202,"num_ops":1,"prepare.queue_time_us":359364,"prepare.run_cpu_time_us":689,"prepare.run_wall_time_us":793761,"replication_time_us":319037,"spinlock_wait_cycles":12672,"thread_start_us":96,"threads_started":1}]]}
W20260921 18:45:08.633226 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.803s user 0.001s sys 0.000s
I20260921 18:45:08.706912 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:08.707052 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:08.707829 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:08.708016 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:08.708217 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:08.708635 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:08.708822 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:08.713254 6100 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=18, attempt_no=0}) took 1033 ms. Trace:
I20260921 18:45:08.713331 6100 rpcz_store.cc:276] 0921 18:45:07.679589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:07.679626 (+ 37us) service_pool.cc:227] Handling call
0921 18:45:08.713241 (+1033615us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:08.312693 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:08.312796 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:08.312801 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:08.312804 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:08.312895 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:08.312905 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:08.312908 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:08.312947 (+ 39us) tablet.cc:699] Row locks acquired
0921 18:45:08.312949 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:08.312986 (+ 37us) write_op.cc:270] Start()
0921 18:45:08.422128 (+109142us) write_op.cc:276] Timestamp: P: 1790016308312920 usec, L: 0
0921 18:45:08.422132 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:08.422361 (+ 229us) log.cc:842] Serialized 354 byte log entry
0921 18:45:08.712802 (+290441us) op_driver.cc:444] REPLICATION: finished
0921 18:45:08.712924 (+ 122us) write_op.cc:301] APPLY: starting
0921 18:45:08.712941 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:08.712964 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:08.712977 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:08.713026 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:08.713048 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:08.713051 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:08.713105 (+ 54us) log.cc:842] Serialized 22 byte log entry
0921 18:45:08.713169 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:08.713178 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:08.713182 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37,"num_ops":1,"prepare.queue_time_us":632724,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":109726,"replication_time_us":290651}]]}
I20260921 18:45:08.869590 6099 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:08.869724 6099 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:08.870272 6099 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:08.870543 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:08.870630 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:08.871012 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:08.871181 6099 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:09.289595 6099 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:09.289743 6099 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:09.290308 6099 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:09.290791 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:09.290872 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:09.291389 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:09.468248 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:09.468425 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:09.468950 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:09.469448 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:09.469578 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:09.470059 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:09.470248 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:09.503530 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.720s user 0.001s sys 0.000s
I20260921 18:45:10.048278 6099 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:10.317600 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:10.317735 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:10.318288 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:10.318658 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:10.318701 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:10.319136 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:10.319306 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:10.342429 6113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=20, attempt_no=0}) took 1627 ms. Trace:
I20260921 18:45:10.342532 6113 rpcz_store.cc:276] 0921 18:45:08.714631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:08.714685 (+ 54us) service_pool.cc:227] Handling call
0921 18:45:10.342408 (+1627723us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:09.046577 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:09.046695 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:09.046700 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:09.046701 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:09.046778 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:09.046786 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:09.046789 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:09.046836 (+ 47us) tablet.cc:699] Row locks acquired
0921 18:45:09.046838 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:09.046876 (+ 38us) write_op.cc:270] Start()
0921 18:45:09.574001 (+527125us) write_op.cc:276] Timestamp: P: 1790016309046847 usec, L: 0
0921 18:45:09.574004 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:09.574152 (+ 148us) log.cc:842] Serialized 354 byte log entry
0921 18:45:10.341699 (+767547us) op_driver.cc:444] REPLICATION: finished
0921 18:45:10.341990 (+ 291us) write_op.cc:301] APPLY: starting
0921 18:45:10.342006 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:10.342033 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:10.342049 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:10.342118 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:10.342145 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:10.342149 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:10.342213 (+ 64us) log.cc:842] Serialized 22 byte log entry
0921 18:45:10.342305 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:10.342318 (+ 13us) write_op.cc:454] Released schema lock
0921 18:45:10.342325 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":230,"num_ops":1,"prepare.queue_time_us":331584,"prepare.run_cpu_time_us":608,"prepare.run_wall_time_us":527663,"raft.queue_time_us":37,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"replication_time_us":767661,"spinlock_wait_cycles":18688,"thread_start_us":104,"threads_started":1}]]}
W20260921 18:45:10.540726 5769 tablet_service.cc:2970] error setting up scanner 0d51fbad268d4c3eb938766eaa209ada with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.491s (maximum allowed 0.100s): new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:10.731184 6099 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:10.731361 6099 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:10.731920 6099 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:10.732230 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:10.732342 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:10.732805 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:10.743552 6113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=21, attempt_no=0}) took 1239 ms. Trace:
I20260921 18:45:10.743705 6113 rpcz_store.cc:276] 0921 18:45:09.503915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:09.503952 (+ 37us) service_pool.cc:227] Handling call
0921 18:45:10.743542 (+1239590us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:09.574266 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:09.574366 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:09.574371 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:09.574375 (+ 4us) tablet.cc:660] Decoding operations
0921 18:45:09.574484 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:09.574496 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:09.574501 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:09.574541 (+ 40us) tablet.cc:699] Row locks acquired
0921 18:45:09.574542 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:09.574580 (+ 38us) write_op.cc:270] Start()
0921 18:45:09.764713 (+190133us) write_op.cc:276] Timestamp: P: 1790016309574543 usec, L: 0
0921 18:45:09.764718 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:09.764982 (+ 264us) log.cc:842] Serialized 354 byte log entry
0921 18:45:10.743167 (+978185us) op_driver.cc:444] REPLICATION: finished
0921 18:45:10.743268 (+ 101us) write_op.cc:301] APPLY: starting
0921 18:45:10.743281 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:10.743301 (+ 20us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:10.743312 (+ 11us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:10.743354 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:10.743375 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:10.743378 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:10.743428 (+ 50us) log.cc:842] Serialized 22 byte log entry
0921 18:45:10.743481 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:10.743488 (+ 7us) write_op.cc:454] Released schema lock
0921 18:45:10.743492 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"num_ops":1,"prepare.queue_time_us":69995,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":190767,"replication_time_us":978429}]]}
I20260921 18:45:10.744412 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 33) took 1089 ms. Trace:
I20260921 18:45:10.744485 5637 rpcz_store.cc:276] 0921 18:45:09.654699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:09.654834 (+ 135us) service_pool.cc:227] Handling call
0921 18:45:09.654995 (+ 161us) tablet_service.cc:2780] Created scanner b3ceeb84bb20444d938494b60561117c for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 3ac2159a860a4c14ab42b911da384e1b
0921 18:45:09.655159 (+ 164us) tablet_service.cc:2920] Creating iterator
0921 18:45:09.655182 (+ 23us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:09.764838 (+109656us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:10.743525 (+978687us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1088331 microseconds
0921 18:45:10.743626 (+ 101us) tablet_service.cc:2945] Iterator created
0921 18:45:10.743919 (+ 293us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:10.743948 (+ 29us) tablet_service.cc:3010] has_more: true
0921 18:45:10.743986 (+ 38us) tablet_service.cc:3027] Continuing scan request
0921 18:45:10.744037 (+ 51us) tablet_service.cc:3091] Found scanner b3ceeb84bb20444d938494b60561117c for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 3ac2159a860a4c14ab42b911da384e1b
0921 18:45:10.744402 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:10.745095 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.091s user 0.001s sys 0.000s
I20260921 18:45:11.102382 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:11.102519 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:11.103016 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:11.103574 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:11.104962 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:11.105398 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:11.440629 6099 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:11.447940 6122 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=22, attempt_no=0}) took 1104 ms. Trace:
I20260921 18:45:11.448016 6122 rpcz_store.cc:276] 0921 18:45:10.343835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:10.343910 (+ 75us) service_pool.cc:227] Handling call
0921 18:45:11.447925 (+1104015us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:10.344405 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:10.344521 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:10.344524 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:10.344524 (+ 0us) tablet.cc:660] Decoding operations
0921 18:45:10.344581 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:10.344587 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:10.344589 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:10.344617 (+ 28us) tablet.cc:699] Row locks acquired
0921 18:45:10.344618 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:10.344645 (+ 27us) write_op.cc:270] Start()
0921 18:45:10.357814 (+ 13169us) write_op.cc:276] Timestamp: P: 1790016310344598 usec, L: 0
0921 18:45:10.357817 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:10.357978 (+ 161us) log.cc:842] Serialized 354 byte log entry
0921 18:45:11.447472 (+1089494us) op_driver.cc:444] REPLICATION: finished
0921 18:45:11.447658 (+ 186us) write_op.cc:301] APPLY: starting
0921 18:45:11.447672 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:11.447691 (+ 19us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:11.447704 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:11.447745 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:11.447764 (+ 19us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:11.447767 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:11.447810 (+ 43us) log.cc:842] Serialized 22 byte log entry
0921 18:45:11.447851 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:11.447872 (+ 21us) write_op.cc:454] Released schema lock
0921 18:45:11.447876 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141,"num_ops":1,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":13641,"replication_time_us":1089634,"thread_start_us":127,"threads_started":2}]]}
I20260921 18:45:11.587199 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:11.595865 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:11.595976 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:11.596290 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:11.596601 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:11.596628 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:11.596971 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:11.597270 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:11.611678 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.716s user 0.001s sys 0.000s
I20260921 18:45:11.649310 6069 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:11.649417 6069 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:11.649876 6069 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:11.650219 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:11.650238 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:11.650636 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:11.650789 6069 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:11.796765 6099 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:11.796906 6099 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:11.797398 6099 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:11.797801 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:11.798610 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:11.798995 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:11.799165 6099 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:12.642817 6127 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:12.643078 6127 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:12.643571 6127 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:12.643975 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:12.644028 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:12.644430 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:12.644625 6127 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:12.646520 6130 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=24, attempt_no=0}) took 1197 ms. Trace:
I20260921 18:45:12.646631 6130 rpcz_store.cc:276] 0921 18:45:11.449250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:11.449317 (+ 67us) service_pool.cc:227] Handling call
0921 18:45:12.646506 (+1197189us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:11.449659 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:11.449727 (+ 68us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:11.449730 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:11.449731 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:11.449782 (+ 51us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:11.449787 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:11.449789 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:11.449814 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:45:11.449815 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:11.449839 (+ 24us) write_op.cc:270] Start()
0921 18:45:12.438958 (+989119us) write_op.cc:276] Timestamp: P: 1790016311449785 usec, L: 0
0921 18:45:12.438961 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:12.439185 (+ 224us) log.cc:842] Serialized 354 byte log entry
0921 18:45:12.645895 (+206710us) op_driver.cc:444] REPLICATION: finished
0921 18:45:12.646178 (+ 283us) write_op.cc:301] APPLY: starting
0921 18:45:12.646192 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:12.646213 (+ 21us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:12.646225 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:12.646279 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:12.646300 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:12.646303 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:12.646346 (+ 43us) log.cc:842] Serialized 22 byte log entry
0921 18:45:12.646430 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:12.646440 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:12.646444 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":233,"mutex_wait_us":18,"num_ops":1,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":565,"prepare.run_wall_time_us":989627,"raft.queue_time_us":93,"raft.run_cpu_time_us":348,"raft.run_wall_time_us":480,"replication_time_us":206912,"spinlock_wait_cycles":438016,"thread_start_us":126,"threads_started":1}]]}
W20260921 18:45:12.647724 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.885s user 0.001s sys 0.000s
I20260921 18:45:12.967795 6126 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:12.967942 6126 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:12.968518 6126 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:12.968806 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:12.968788 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:12.969635 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:13.042193 6127 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:13.042332 6127 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:13.042819 6127 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:13.043259 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:13.043260 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:13.043766 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:13.043982 6127 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:13.067549 6130 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=25, attempt_no=0}) took 1455 ms. Trace:
I20260921 18:45:13.067636 6130 rpcz_store.cc:276] 0921 18:45:11.611646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:11.611686 (+ 40us) service_pool.cc:227] Handling call
0921 18:45:13.067534 (+1455848us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:12.439315 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:12.439431 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:12.439436 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:12.439438 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:12.439512 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:12.439520 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:12.439523 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:12.439561 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:45:12.439563 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:12.439594 (+ 31us) write_op.cc:270] Start()
0921 18:45:12.652715 (+213121us) write_op.cc:276] Timestamp: P: 1790016312439541 usec, L: 0
0921 18:45:12.652717 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:45:12.652845 (+ 128us) log.cc:842] Serialized 354 byte log entry
0921 18:45:13.067130 (+414285us) op_driver.cc:444] REPLICATION: finished
0921 18:45:13.067233 (+ 103us) write_op.cc:301] APPLY: starting
0921 18:45:13.067247 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:13.067269 (+ 22us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:13.067280 (+ 11us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:13.067324 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:13.067348 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:13.067350 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:13.067402 (+ 52us) log.cc:842] Serialized 22 byte log entry
0921 18:45:13.067444 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:13.067453 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:13.067457 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"num_ops":1,"prepare.queue_time_us":827331,"prepare.run_cpu_time_us":538,"prepare.run_wall_time_us":213609,"replication_time_us":414375}]]}
I20260921 18:45:13.225306 6127 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:13.225430 6127 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:13.225844 6127 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:13.226197 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:13.226322 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:13.226838 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:13.227401 6127 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:13.690122 6126 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:13.888017 6126 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:13.888145 6126 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:13.888756 6126 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:13.889184 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:13.889402 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:13.889809 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:13.928947 6126 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:13.959722 6136 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:13.959939 6136 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:13.960530 6136 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:13.960799 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:13.960952 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:13.961373 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:13.961869 6136 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:13.970633 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.580s user 0.001s sys 0.000s
I20260921 18:45:15.129616 6141 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:15.129750 6141 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:15.130329 6141 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:15.130712 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:15.130712 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:15.131114 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:15.131274 6141 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:15.165730 6148 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=28, attempt_no=0}) took 1925 ms. Trace:
I20260921 18:45:15.165799 6148 rpcz_store.cc:276] 0921 18:45:13.240205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:13.240269 (+ 64us) service_pool.cc:227] Handling call
0921 18:45:15.165716 (+1925447us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:13.854267 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:13.854363 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:13.854367 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:13.854369 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:13.854441 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:13.854469 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:13.854474 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:13.854511 (+ 37us) tablet.cc:699] Row locks acquired
0921 18:45:13.854512 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:13.854542 (+ 30us) write_op.cc:270] Start()
0921 18:45:14.534692 (+680150us) write_op.cc:276] Timestamp: P: 1790016313854490 usec, L: 0
0921 18:45:14.534697 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:14.534890 (+ 193us) log.cc:842] Serialized 354 byte log entry
0921 18:45:15.165178 (+630288us) op_driver.cc:444] REPLICATION: finished
0921 18:45:15.165365 (+ 187us) write_op.cc:301] APPLY: starting
0921 18:45:15.165377 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:15.165400 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:15.165412 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:15.165457 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:15.165532 (+ 75us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:15.165540 (+ 8us) write_op.cc:312] APPLY: finished
0921 18:45:15.165597 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:45:15.165650 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:15.165659 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:15.165663 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":143,"num_ops":1,"prepare.queue_time_us":613756,"prepare.run_cpu_time_us":671,"prepare.run_wall_time_us":680753,"raft.queue_time_us":104,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":150,"replication_time_us":630458,"spinlock_wait_cycles":23296,"thread_start_us":68,"threads_started":1}]]}
I20260921 18:45:15.166661 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 46) took 1045 ms. Trace:
I20260921 18:45:15.166738 5637 rpcz_store.cc:276] 0921 18:45:14.121693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:14.121733 (+ 40us) service_pool.cc:227] Handling call
0921 18:45:14.121889 (+ 156us) tablet_service.cc:2780] Created scanner b1d0b0a543324003af6f127a0d50a759 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 4ce89aa56c894f1b85f973336cac4064
0921 18:45:14.122049 (+ 160us) tablet_service.cc:2920] Creating iterator
0921 18:45:14.122078 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:14.534807 (+412729us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:15.165703 (+630896us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1043612 microseconds
0921 18:45:15.165787 (+ 84us) tablet_service.cc:2945] Iterator created
0921 18:45:15.166136 (+ 349us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:15.166163 (+ 27us) tablet_service.cc:3010] has_more: true
0921 18:45:15.166197 (+ 34us) tablet_service.cc:3027] Continuing scan request
0921 18:45:15.166268 (+ 71us) tablet_service.cc:3091] Found scanner b1d0b0a543324003af6f127a0d50a759 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 4ce89aa56c894f1b85f973336cac4064
0921 18:45:15.166650 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:15.167163 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.046s user 0.000s sys 0.000s
I20260921 18:45:15.260712 6141 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:15.260854 6141 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:15.261406 6141 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:15.261734 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:15.261865 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:15.262243 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:15.262434 6141 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:15.286329 6148 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=29, attempt_no=0}) took 1315 ms. Trace:
I20260921 18:45:15.286437 6148 rpcz_store.cc:276] 0921 18:45:13.970391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:13.970429 (+ 38us) service_pool.cc:227] Handling call
0921 18:45:15.286298 (+1315869us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:14.535043 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:14.535147 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:14.535151 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:14.535153 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:14.535232 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:14.535245 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:14.535250 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:14.535306 (+ 56us) tablet.cc:699] Row locks acquired
0921 18:45:14.535308 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:14.535341 (+ 33us) write_op.cc:270] Start()
0921 18:45:15.160464 (+625123us) write_op.cc:276] Timestamp: P: 1790016314535284 usec, L: 0
0921 18:45:15.160468 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:15.160602 (+ 134us) log.cc:842] Serialized 354 byte log entry
0921 18:45:15.285848 (+125246us) op_driver.cc:444] REPLICATION: finished
0921 18:45:15.285932 (+ 84us) write_op.cc:301] APPLY: starting
0921 18:45:15.285947 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:15.285973 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:15.285990 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:15.286045 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:15.286091 (+ 46us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:15.286097 (+ 6us) write_op.cc:312] APPLY: finished
0921 18:45:15.286157 (+ 60us) log.cc:842] Serialized 22 byte log entry
0921 18:45:15.286216 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:15.286230 (+ 14us) write_op.cc:454] Released schema lock
0921 18:45:15.286234 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"num_ops":1,"prepare.queue_time_us":564281,"prepare.run_cpu_time_us":541,"prepare.run_wall_time_us":625609,"replication_time_us":125359}]]}
I20260921 18:45:15.466408 6142 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:15.466557 6142 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:15.467057 6142 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:15.467344 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:15.467545 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:15.467905 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:15.790869 6142 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:16.150636 5769 tablet_service.cc:2970] error setting up scanner 4c38ecc018ab42418b0bb80237c45d29 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790016315150417 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790016314535284 usec, L: 0 Physical time difference: 0.615s: new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:16.590523 6142 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:16.590638 6142 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:16.591055 6142 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:16.591495 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:16.591621 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:16.592134 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:16.592491 6142 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:16.598476 6155 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=30, attempt_no=0}) took 1431 ms. Trace:
I20260921 18:45:16.598585 6155 rpcz_store.cc:276] 0921 18:45:15.167104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:15.167161 (+ 57us) service_pool.cc:227] Handling call
0921 18:45:16.598463 (+1431302us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:15.167526 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:15.167605 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:15.167607 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:15.167608 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:15.167656 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:15.167662 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:15.167663 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:15.167688 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:45:15.167689 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:15.167712 (+ 23us) write_op.cc:270] Start()
0921 18:45:16.104866 (+937154us) write_op.cc:276] Timestamp: P: 1790016315167681 usec, L: 0
0921 18:45:16.104870 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:16.105085 (+ 215us) log.cc:842] Serialized 354 byte log entry
0921 18:45:16.597886 (+492801us) op_driver.cc:444] REPLICATION: finished
0921 18:45:16.598074 (+ 188us) write_op.cc:301] APPLY: starting
0921 18:45:16.598087 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:16.598116 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:16.598128 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:16.598170 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:16.598197 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:16.598200 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:16.598297 (+ 97us) log.cc:842] Serialized 22 byte log entry
0921 18:45:16.598387 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:16.598398 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:16.598402 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":136,"num_ops":1,"prepare.queue_time_us":26,"prepare.run_cpu_time_us":563,"prepare.run_wall_time_us":937638,"raft.queue_time_us":40,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"replication_time_us":492994,"spinlock_wait_cycles":43776,"thread_start_us":67,"threads_started":1}]]}
I20260921 18:45:16.599326 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 49) took 1281 ms. Trace:
I20260921 18:45:16.599393 5637 rpcz_store.cc:276] 0921 18:45:15.318163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:15.318210 (+ 47us) service_pool.cc:227] Handling call
0921 18:45:15.318329 (+ 119us) tablet_service.cc:2780] Created scanner 1893d106c80b47efa4549e62c27fc3a3 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 7741bd923dad493091f3128704e1e6f7
0921 18:45:15.318472 (+ 143us) tablet_service.cc:2920] Creating iterator
0921 18:45:15.318491 (+ 19us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:16.104993 (+786502us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:16.598411 (+493418us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1279908 microseconds
0921 18:45:16.598487 (+ 76us) tablet_service.cc:2945] Iterator created
0921 18:45:16.598807 (+ 320us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:16.598832 (+ 25us) tablet_service.cc:3010] has_more: true
0921 18:45:16.598869 (+ 37us) tablet_service.cc:3027] Continuing scan request
0921 18:45:16.598932 (+ 63us) tablet_service.cc:3091] Found scanner 1893d106c80b47efa4549e62c27fc3a3 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 7741bd923dad493091f3128704e1e6f7
0921 18:45:16.599315 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:16.599817 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.282s user 0.000s sys 0.000s
I20260921 18:45:16.852439 6152 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:16.852597 6152 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:16.853114 6152 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:16.853365 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:16.853623 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:16.854295 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:16.854430 6152 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:17.300544 6152 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:17.300695 6152 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:17.301231 6152 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:17.301635 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:17.301635 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:17.302129 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:17.302428 6152 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:17.324355 6160 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=31, attempt_no=0}) took 2036 ms. Trace:
I20260921 18:45:17.324434 6160 rpcz_store.cc:276] 0921 18:45:15.287859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:15.287930 (+ 71us) service_pool.cc:227] Handling call
0921 18:45:17.324338 (+2036408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:16.105190 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:16.105286 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:16.105290 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:16.105293 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:16.105370 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:16.105378 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:16.105381 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:16.105419 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:45:16.105421 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:16.105453 (+ 32us) write_op.cc:270] Start()
0921 18:45:17.091594 (+986141us) write_op.cc:276] Timestamp: P: 1790016316105393 usec, L: 0
0921 18:45:17.091597 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:17.091799 (+ 202us) log.cc:842] Serialized 354 byte log entry
0921 18:45:17.323523 (+231724us) op_driver.cc:444] REPLICATION: finished
0921 18:45:17.323760 (+ 237us) write_op.cc:301] APPLY: starting
0921 18:45:17.323773 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:17.323800 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:17.323821 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:17.323862 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:17.323905 (+ 43us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:17.323909 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:17.323971 (+ 62us) log.cc:842] Serialized 22 byte log entry
0921 18:45:17.324061 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:17.324070 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:17.324081 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":173,"num_ops":1,"prepare.queue_time_us":816920,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":986690,"raft.queue_time_us":111,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":231900,"spinlock_wait_cycles":1408,"thread_start_us":78,"threads_started":1}]]}
W20260921 18:45:17.325754 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.575s user 0.001s sys 0.000s
I20260921 18:45:17.922849 6152 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:17.922983 6152 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:17.923470 6152 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:17.923815 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:17.923815 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:17.924214 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:17.924388 6152 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:17.963454 6164 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=32, attempt_no=0}) took 1363 ms. Trace:
I20260921 18:45:17.963591 6164 rpcz_store.cc:276] 0921 18:45:16.599961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:16.600015 (+ 54us) service_pool.cc:227] Handling call
0921 18:45:17.963437 (+1363422us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:17.091907 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:17.092035 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:17.092040 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:17.092044 (+ 4us) tablet.cc:660] Decoding operations
0921 18:45:17.092122 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:17.092131 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:17.092134 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:17.092174 (+ 40us) tablet.cc:699] Row locks acquired
0921 18:45:17.092175 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:17.092214 (+ 39us) write_op.cc:270] Start()
0921 18:45:17.578345 (+486131us) write_op.cc:276] Timestamp: P: 1790016317092150 usec, L: 0
0921 18:45:17.578348 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:17.578528 (+ 180us) log.cc:842] Serialized 354 byte log entry
0921 18:45:17.962824 (+384296us) op_driver.cc:444] REPLICATION: finished
0921 18:45:17.963050 (+ 226us) write_op.cc:301] APPLY: starting
0921 18:45:17.963066 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:17.963103 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:17.963119 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:17.963172 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:17.963203 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:17.963207 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:17.963263 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:45:17.963354 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:17.963367 (+ 13us) write_op.cc:454] Released schema lock
0921 18:45:17.963372 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":151,"num_ops":1,"prepare.queue_time_us":491608,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":486717,"raft.queue_time_us":37,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":4,"replication_time_us":384452,"spinlock_wait_cycles":22400,"thread_start_us":63,"threads_started":1}]]}
I20260921 18:45:18.091251 6159 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:18.091401 6159 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:18.091929 6159 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:18.092253 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:18.092546 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:18.092952 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:18.169523 6159 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:19.143775 6159 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:19.143911 6159 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:19.144505 6159 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:19.144984 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:19.145000 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:19.145691 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:19.182607 6173 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=33, attempt_no=0}) took 1856 ms. Trace:
I20260921 18:45:19.182754 6173 rpcz_store.cc:276] 0921 18:45:17.325679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:17.325719 (+ 40us) service_pool.cc:227] Handling call
0921 18:45:19.182590 (+1856871us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:17.578667 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:17.578766 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:17.578771 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:17.578772 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:17.578845 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:17.578861 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:17.578863 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:17.578897 (+ 34us) tablet.cc:699] Row locks acquired
0921 18:45:17.578899 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:17.578948 (+ 49us) write_op.cc:270] Start()
0921 18:45:18.440072 (+861124us) write_op.cc:276] Timestamp: P: 1790016317578867 usec, L: 0
0921 18:45:18.440076 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:18.440280 (+ 204us) log.cc:842] Serialized 354 byte log entry
0921 18:45:19.182024 (+741744us) op_driver.cc:444] REPLICATION: finished
0921 18:45:19.182264 (+ 240us) write_op.cc:301] APPLY: starting
0921 18:45:19.182278 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:19.182304 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:19.182317 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:19.182375 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:19.182396 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:19.182399 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:19.182455 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:45:19.182510 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:19.182519 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:19.182523 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":189,"num_ops":1,"prepare.queue_time_us":252645,"prepare.run_cpu_time_us":637,"prepare.run_wall_time_us":861693,"raft.queue_time_us":76,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":741914,"thread_start_us":85,"threads_started":1}]]}
I20260921 18:45:19.183537 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 55) took 1067 ms. Trace:
I20260921 18:45:19.183614 5637 rpcz_store.cc:276] 0921 18:45:18.115605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:18.115675 (+ 70us) service_pool.cc:227] Handling call
0921 18:45:18.115815 (+ 140us) tablet_service.cc:2780] Created scanner 14bafd2a928441aaa0a68a41968c9b95 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 4992211e04494b4d916ac2e6d31fd2b3
0921 18:45:18.116024 (+ 209us) tablet_service.cc:2920] Creating iterator
0921 18:45:18.116042 (+ 18us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:18.440177 (+324135us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:19.182557 (+742380us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1066501 microseconds
0921 18:45:19.182641 (+ 84us) tablet_service.cc:2945] Iterator created
0921 18:45:19.182993 (+ 352us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:19.183021 (+ 28us) tablet_service.cc:3010] has_more: true
0921 18:45:19.183076 (+ 55us) tablet_service.cc:3027] Continuing scan request
0921 18:45:19.183126 (+ 50us) tablet_service.cc:3091] Found scanner 14bafd2a928441aaa0a68a41968c9b95 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 4992211e04494b4d916ac2e6d31fd2b3
0921 18:45:19.183499 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:19.184302 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.069s user 0.001s sys 0.000s
I20260921 18:45:19.262099 6168 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:19.262259 6168 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:19.262844 6168 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:19.263193 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:19.263208 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:19.263760 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:19.326310 6159 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:19.335764 6173 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=34, attempt_no=0}) took 1370 ms. Trace:
I20260921 18:45:19.335837 6173 rpcz_store.cc:276] 0921 18:45:17.964901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:17.965003 (+ 102us) service_pool.cc:227] Handling call
0921 18:45:19.335751 (+1370748us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:18.440394 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:18.440492 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:18.440497 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:18.440499 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:18.440583 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:18.440592 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:18.440594 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:18.440632 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:45:18.440636 (+ 4us) write_op.cc:260] PREPARE: finished
0921 18:45:18.440671 (+ 35us) write_op.cc:270] Start()
0921 18:45:19.074814 (+634143us) write_op.cc:276] Timestamp: P: 1790016318440630 usec, L: 0
0921 18:45:19.074819 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:19.075008 (+ 189us) log.cc:842] Serialized 354 byte log entry
0921 18:45:19.335197 (+260189us) op_driver.cc:444] REPLICATION: finished
0921 18:45:19.335378 (+ 181us) write_op.cc:301] APPLY: starting
0921 18:45:19.335411 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:19.335437 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:19.335453 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:19.335511 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:19.335542 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:19.335545 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:19.335626 (+ 81us) log.cc:842] Serialized 22 byte log entry
0921 18:45:19.335679 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:19.335688 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:19.335692 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":138,"num_ops":1,"prepare.queue_time_us":475056,"prepare.run_cpu_time_us":607,"prepare.run_wall_time_us":634674,"replication_time_us":260351}]]}
I20260921 18:45:19.944443 6178 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:19.944590 6178 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:19.945171 6178 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:19.945552 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:19.945552 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:19.946000 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:19.946165 6178 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:19.952037 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.617s user 0.001s sys 0.000s
I20260921 18:45:20.059481 6168 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:20.749670 6168 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:20.749815 6168 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:20.750326 6168 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:20.750797 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:20.750811 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:20.751269 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:21.116987 6168 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:21.155706 6188 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=36, attempt_no=0}) took 1818 ms. Trace:
I20260921 18:45:21.155802 6188 rpcz_store.cc:276] 0921 18:45:19.337030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:19.337104 (+ 74us) service_pool.cc:227] Handling call
0921 18:45:21.155687 (+1818583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:19.887355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:19.887451 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:19.887455 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:19.887457 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:19.887556 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:19.887565 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:19.887568 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:19.887616 (+ 48us) tablet.cc:699] Row locks acquired
0921 18:45:19.887618 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:19.887653 (+ 35us) write_op.cc:270] Start()
0921 18:45:20.387791 (+500138us) write_op.cc:276] Timestamp: P: 1790016319887611 usec, L: 0
0921 18:45:20.387796 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:20.388032 (+ 236us) log.cc:842] Serialized 354 byte log entry
0921 18:45:21.155080 (+767048us) op_driver.cc:444] REPLICATION: finished
0921 18:45:21.155295 (+ 215us) write_op.cc:301] APPLY: starting
0921 18:45:21.155309 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:21.155332 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:21.155345 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:21.155442 (+ 97us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:21.155465 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:21.155468 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:21.155526 (+ 58us) log.cc:842] Serialized 22 byte log entry
0921 18:45:21.155602 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:21.155616 (+ 14us) write_op.cc:454] Released schema lock
0921 18:45:21.155620 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":162,"num_ops":1,"prepare.queue_time_us":549924,"prepare.run_cpu_time_us":666,"prepare.run_wall_time_us":500738,"replication_time_us":767257,"thread_start_us":76,"threads_started":1}]]}
I20260921 18:45:21.156510 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 60) took 1053 ms. Trace:
I20260921 18:45:21.156589 5637 rpcz_store.cc:276] 0921 18:45:20.103172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:20.103237 (+ 65us) service_pool.cc:227] Handling call
0921 18:45:20.103364 (+ 127us) tablet_service.cc:2780] Created scanner 741b22669fa44f45b8c677f33573c93c for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is e411b71d8b1e43689aaf8e4e794b540a
0921 18:45:20.103549 (+ 185us) tablet_service.cc:2920] Creating iterator
0921 18:45:20.103576 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:20.387905 (+284329us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:21.155630 (+767725us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1052043 microseconds
0921 18:45:21.155711 (+ 81us) tablet_service.cc:2945] Iterator created
0921 18:45:21.156038 (+ 327us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:21.156062 (+ 24us) tablet_service.cc:3010] has_more: true
0921 18:45:21.156095 (+ 33us) tablet_service.cc:3027] Continuing scan request
0921 18:45:21.156148 (+ 53us) tablet_service.cc:3091] Found scanner 741b22669fa44f45b8c677f33573c93c for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is e411b71d8b1e43689aaf8e4e794b540a
0921 18:45:21.156500 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:21.157011 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.054s user 0.001s sys 0.000s
I20260921 18:45:21.215870 6178 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:21.216068 6178 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:21.216609 6178 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:21.216975 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:21.217046 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:21.217442 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:21.334398 6178 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:21.367640 6188 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=37, attempt_no=0}) took 1415 ms. Trace:
I20260921 18:45:21.367733 6188 rpcz_store.cc:276] 0921 18:45:19.952020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:19.952061 (+ 41us) service_pool.cc:227] Handling call
0921 18:45:21.367602 (+1415541us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:20.388128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:20.388223 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:20.388227 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:20.388228 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:20.388287 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:20.388294 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:20.388296 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:20.388353 (+ 57us) tablet.cc:699] Row locks acquired
0921 18:45:20.388356 (+ 3us) write_op.cc:260] PREPARE: finished
0921 18:45:20.388395 (+ 39us) write_op.cc:270] Start()
0921 18:45:21.045592 (+657197us) write_op.cc:276] Timestamp: P: 1790016320388360 usec, L: 0
0921 18:45:21.045596 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:21.045806 (+ 210us) log.cc:842] Serialized 354 byte log entry
0921 18:45:21.367202 (+321396us) op_driver.cc:444] REPLICATION: finished
0921 18:45:21.367293 (+ 91us) write_op.cc:301] APPLY: starting
0921 18:45:21.367316 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:21.367338 (+ 22us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:21.367352 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:21.367398 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:21.367420 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:21.367422 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:21.367492 (+ 70us) log.cc:842] Serialized 22 byte log entry
0921 18:45:21.367534 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:21.367542 (+ 8us) write_op.cc:454] Released schema lock
0921 18:45:21.367545 (+ 3us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":44,"num_ops":1,"prepare.queue_time_us":435728,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":657741,"replication_time_us":321581}]]}
I20260921 18:45:21.487509 6178 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:21.487653 6178 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:21.488215 6178 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:21.488628 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:21.488647 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:21.489077 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:21.489295 6178 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:21.760469 5769 tablet_service.cc:2970] error setting up scanner f81f8fe4cabf4c699a64254b10e9e386 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1790016320760192 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1790016319887611 usec, L: 0 Physical time difference: 0.873s: new_scan_request { tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\326\370\305\325\006\"3\n\005slave\022*\n d44aff99241b41e280681bc2570e93d2\020\001\030\001 \001(\001" signature: "<redacted>"".@^?\353a\2204fk\350# K\312x\373\0051\251s\024\335\325s\001\023\323\340\343\036\210\257\337\220>\320j\236\322\227\356\323w\340.\377\202i\307*\205\\w\260\356bv\214Q\300]q\364" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "5e5131bcb8044a9d89fa1f0216b6352e"
I20260921 18:45:21.792317 6168 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:21.792478 6168 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:21.792992 6168 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:21.793314 5939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:21.793342 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:21.793777 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:21.939882 6168 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:22.762871 6168 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:22.762988 6168 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:22.763465 6168 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:22.763787 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:22.764142 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:22.764465 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:22.781960 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:22.782076 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:22.782634 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:22.783010 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:22.783021 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:22.783468 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:22.784085 6201 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=39, attempt_no=0}) took 1415 ms. Trace:
I20260921 18:45:22.784185 6201 rpcz_store.cc:276] 0921 18:45:21.369071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:21.369131 (+ 60us) service_pool.cc:227] Handling call
0921 18:45:22.784073 (+1414942us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:21.369545 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:21.369617 (+ 72us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:21.369620 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:21.369621 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:21.369671 (+ 50us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:21.369676 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:21.369678 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:21.369704 (+ 26us) tablet.cc:699] Row locks acquired
0921 18:45:21.369704 (+ 0us) write_op.cc:260] PREPARE: finished
0921 18:45:21.369729 (+ 25us) write_op.cc:270] Start()
0921 18:45:22.026869 (+657140us) write_op.cc:276] Timestamp: P: 1790016321369698 usec, L: 0
0921 18:45:22.026873 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:22.027095 (+ 222us) log.cc:842] Serialized 354 byte log entry
0921 18:45:22.783200 (+756105us) op_driver.cc:444] REPLICATION: finished
0921 18:45:22.783724 (+ 524us) write_op.cc:301] APPLY: starting
0921 18:45:22.783739 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:22.783765 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:22.783779 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:22.783833 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:22.783856 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:22.783858 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:22.783906 (+ 48us) log.cc:842] Serialized 22 byte log entry
0921 18:45:22.783982 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:22.783994 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:22.783999 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":466,"num_ops":1,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":536,"prepare.run_wall_time_us":657611,"replication_time_us":756307,"thread_start_us":89,"threads_started":1}]]}
I20260921 18:45:22.784909 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 65) took 1136 ms. Trace:
I20260921 18:45:22.784974 5637 rpcz_store.cc:276] 0921 18:45:21.648117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:21.648176 (+ 59us) service_pool.cc:227] Handling call
0921 18:45:21.648316 (+ 140us) tablet_service.cc:2780] Created scanner 985641c2156141b5a1035e6e0340ef0e for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is cf77ba04622c42f8bca8551e208a366b
0921 18:45:21.648470 (+ 154us) tablet_service.cc:2920] Creating iterator
0921 18:45:21.648493 (+ 23us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:22.026988 (+378495us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:22.784007 (+757019us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1135504 microseconds
0921 18:45:22.784080 (+ 73us) tablet_service.cc:2945] Iterator created
0921 18:45:22.784400 (+ 320us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:22.784429 (+ 29us) tablet_service.cc:3010] has_more: true
0921 18:45:22.784456 (+ 27us) tablet_service.cc:3027] Continuing scan request
0921 18:45:22.784525 (+ 69us) tablet_service.cc:3091] Found scanner 985641c2156141b5a1035e6e0340ef0e for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is cf77ba04622c42f8bca8551e208a366b
0921 18:45:22.784899 (+ 374us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:22.785518 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.138s user 0.001s sys 0.000s
I20260921 18:45:23.185282 6168 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:23.191715 6201 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=40, attempt_no=0}) took 1694 ms. Trace:
I20260921 18:45:23.191813 6201 rpcz_store.cc:276] 0921 18:45:21.496925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:21.496992 (+ 67us) service_pool.cc:227] Handling call
0921 18:45:23.191701 (+1694709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:22.027187 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:22.027289 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:22.027296 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:45:22.027299 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:22.027372 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:22.027381 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:22.027383 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:22.027424 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:45:22.027425 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:22.027461 (+ 36us) write_op.cc:270] Start()
0921 18:45:22.434638 (+407177us) write_op.cc:276] Timestamp: P: 1790016322027422 usec, L: 0
0921 18:45:22.434642 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:22.434845 (+ 203us) log.cc:842] Serialized 354 byte log entry
0921 18:45:23.191260 (+756415us) op_driver.cc:444] REPLICATION: finished
0921 18:45:23.191371 (+ 111us) write_op.cc:301] APPLY: starting
0921 18:45:23.191386 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:23.191408 (+ 22us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:23.191422 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:23.191472 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:23.191501 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:23.191504 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:23.191559 (+ 55us) log.cc:842] Serialized 22 byte log entry
0921 18:45:23.191625 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:23.191635 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:23.191639 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":529854,"prepare.run_cpu_time_us":633,"prepare.run_wall_time_us":407718,"replication_time_us":756580}]]}
I20260921 18:45:23.468539 6168 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:23.468688 6168 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:23.469202 6168 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:23.469594 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:23.469604 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:23.470064 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
W20260921 18:45:23.510205 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.574s user 0.001s sys 0.000s
I20260921 18:45:23.727963 6168 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:23.759459 6194 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:24.611064 6168 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:24.611205 6168 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:24.611735 6168 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:24.612079 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:24.612358 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:24.613036 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:24.661964 6210 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=43, attempt_no=0}) took 1151 ms. Trace:
I20260921 18:45:24.662128 6210 rpcz_store.cc:276] 0921 18:45:23.510202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:23.510265 (+ 63us) service_pool.cc:227] Handling call
0921 18:45:24.661944 (+1151679us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:23.510614 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:23.510677 (+ 63us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:23.510680 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:23.510681 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:23.510732 (+ 51us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:23.510737 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:23.510739 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:23.510764 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:45:23.510765 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:23.510788 (+ 23us) write_op.cc:270] Start()
0921 18:45:23.715932 (+205144us) write_op.cc:276] Timestamp: P: 1790016323510746 usec, L: 0
0921 18:45:23.715936 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:23.716155 (+ 219us) log.cc:842] Serialized 354 byte log entry
0921 18:45:24.661343 (+945188us) op_driver.cc:444] REPLICATION: finished
0921 18:45:24.661591 (+ 248us) write_op.cc:301] APPLY: starting
0921 18:45:24.661608 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:24.661632 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:24.661648 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:24.661699 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:24.661722 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:24.661725 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:24.661783 (+ 58us) log.cc:842] Serialized 22 byte log entry
0921 18:45:24.661840 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:24.661850 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:24.661854 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":193,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":526,"prepare.run_wall_time_us":205592,"replication_time_us":945376,"thread_start_us":99,"threads_started":1}]]}
I20260921 18:45:24.662855 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 70) took 1001 ms. Trace:
I20260921 18:45:24.662936 5637 rpcz_store.cc:276] 0921 18:45:23.661343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:23.661403 (+ 60us) service_pool.cc:227] Handling call
0921 18:45:23.661589 (+ 186us) tablet_service.cc:2780] Created scanner ebb7030157bf4f4c8d706e90a5d75e76 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 647a3204d8584fc7bb908c5df402ba9d
0921 18:45:23.661751 (+ 162us) tablet_service.cc:2920] Creating iterator
0921 18:45:23.661774 (+ 23us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:23.716070 (+ 54296us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:24.661907 (+945837us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1000100 microseconds
0921 18:45:24.661995 (+ 88us) tablet_service.cc:2945] Iterator created
0921 18:45:24.662359 (+ 364us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:24.662382 (+ 23us) tablet_service.cc:3010] has_more: true
0921 18:45:24.662406 (+ 24us) tablet_service.cc:3027] Continuing scan request
0921 18:45:24.662474 (+ 68us) tablet_service.cc:3091] Found scanner ebb7030157bf4f4c8d706e90a5d75e76 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 647a3204d8584fc7bb908c5df402ba9d
0921 18:45:24.662847 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:24.663437 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.003s user 0.001s sys 0.000s
I20260921 18:45:25.019642 6168 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:25.065008 6210 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=44, attempt_no=0}) took 1319 ms. Trace:
I20260921 18:45:25.065094 6210 rpcz_store.cc:276] 0921 18:45:23.745270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:23.745332 (+ 62us) service_pool.cc:227] Handling call
0921 18:45:25.064995 (+1319663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:23.745707 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:23.745788 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:23.745790 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:23.745791 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:23.745844 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:23.745849 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:23.745851 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:23.745880 (+ 29us) tablet.cc:699] Row locks acquired
0921 18:45:23.745881 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:23.745906 (+ 25us) write_op.cc:270] Start()
0921 18:45:24.440041 (+694135us) write_op.cc:276] Timestamp: P: 1790016323745841 usec, L: 0
0921 18:45:24.440045 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:24.440220 (+ 175us) log.cc:842] Serialized 354 byte log entry
0921 18:45:25.064549 (+624329us) op_driver.cc:444] REPLICATION: finished
0921 18:45:25.064637 (+ 88us) write_op.cc:301] APPLY: starting
0921 18:45:25.064652 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:25.064682 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:25.064694 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:25.064740 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:25.064763 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:25.064766 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:25.064849 (+ 83us) log.cc:842] Serialized 22 byte log entry
0921 18:45:25.064909 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:25.064923 (+ 14us) write_op.cc:454] Released schema lock
0921 18:45:25.064928 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":504,"prepare.run_wall_time_us":694570,"replication_time_us":624481}]]}
I20260921 18:45:25.151808 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:25.151929 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:25.152374 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:25.152724 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:25.153278 5939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.129:58825 (request call id 6099) took 1392 ms. Trace:
I20260921 18:45:25.153604 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:25.154035 5876 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:25.154203 6194 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:25.153366 5939 rpcz_store.cc:276] 0921 18:45:23.760562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:23.760615 (+ 53us) service_pool.cc:227] Handling call
0921 18:45:23.760785 (+ 170us) raft_consensus.cc:1457] Updating replica for 3 ops
0921 18:45:23.760873 (+ 88us) raft_consensus.cc:1520] Early marking committed up to index 42
0921 18:45:23.760994 (+ 121us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0921 18:45:23.761081 (+ 87us) write_op.cc:270] Start()
0921 18:45:24.350218 (+589137us) write_op.cc:276] Timestamp: P: 1790016322785994 usec, L: 0
0921 18:45:24.350523 (+ 305us) write_op.cc:270] Start()
0921 18:45:24.361640 (+ 11117us) write_op.cc:276] Timestamp: P: 1790016323345642 usec, L: 0
0921 18:45:24.361785 (+ 145us) write_op.cc:270] Start()
0921 18:45:25.150950 (+789165us) write_op.cc:276] Timestamp: P: 1790016323510746 usec, L: 0
0921 18:45:25.151408 (+ 458us) log.cc:842] Serialized 1062 byte log entry
0921 18:45:25.151526 (+ 118us) raft_consensus.cc:1631] Marking committed up to 44
0921 18:45:25.151761 (+ 235us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:45:25.151789 (+ 28us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:45:25.153225 (+ 1436us) raft_consensus.cc:1671] finished
0921 18:45:25.153228 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:45:25.153267 (+ 39us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:24.350507 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:24.350603 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:24.350607 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:24.350608 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:24.350670 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:24.350680 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:24.350682 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:24.350716 (+ 34us) tablet.cc:699] Row locks acquired
0921 18:45:24.350717 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:25.151555 (+800838us) op_driver.cc:444] REPLICATION: finished
0921 18:45:25.151991 (+ 436us) write_op.cc:301] APPLY: starting
0921 18:45:25.152004 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:25.152031 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:25.152032 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:25.152080 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:25.152132 (+ 52us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:25.152149 (+ 17us) write_op.cc:312] APPLY: finished
0921 18:45:25.152210 (+ 61us) log.cc:842] Serialized 22 byte log entry
0921 18:45:25.152251 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:25.152261 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:25.152265 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0921 18:45:25.152324 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:25.152327 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0921 18:45:24.361815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:24.361886 (+ 71us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:24.361888 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:24.361890 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:24.361932 (+ 42us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:24.361939 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:24.361941 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:24.361967 (+ 26us) tablet.cc:699] Row locks acquired
0921 18:45:24.361968 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:25.151671 (+789703us) op_driver.cc:444] REPLICATION: finished
0921 18:45:25.152014 (+ 343us) write_op.cc:301] APPLY: starting
0921 18:45:25.152027 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:25.152050 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:25.152051 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:25.152098 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:25.152130 (+ 32us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:25.152148 (+ 18us) write_op.cc:312] APPLY: finished
0921 18:45:25.152201 (+ 53us) log.cc:842] Serialized 22 byte log entry
0921 18:45:25.152231 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:25.152240 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:25.152244 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0921 18:45:25.152298 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:25.152300 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0921 18:45:25.151291 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:25.151434 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:25.151439 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:25.151440 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:25.151521 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:25.151534 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:25.151537 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:25.151579 (+ 42us) tablet.cc:699] Row locks acquired
0921 18:45:25.151581 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":118,"child_traces":[["op",{"apply.queue_time_us":396,"apply.run_cpu_time_us":368,"apply.run_wall_time_us":368,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":176,"prepare.run_cpu_time_us":294,"prepare.run_wall_time_us":295,"replication_time_us":1390461,"spinlock_wait_cycles":11264,"thread_start_us":127,"threads_started":2}],["op",{"apply.queue_time_us":300,"apply.run_cpu_time_us":324,"apply.run_wall_time_us":324,"num_ops":1,"partition_lock_wait_us":3,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":206,"prepare.run_wall_time_us":207,"replication_time_us":801140,"thread_start_us":50,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":214,"prepare.run_cpu_time_us":364,"prepare.run_wall_time_us":420,"thread_start_us":92,"threads_started":1}]]}
W20260921 18:45:25.645520 6029 scanner-internal.cc:458] Time spent opening tablet: real 26.811s user 0.006s sys 0.000s
I20260921 18:45:25.862641 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:25.862774 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:25.863323 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:25.863641 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:25.863821 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:25.864238 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:26.452507 6215 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:26.452656 6215 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:26.453271 6215 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:26.453635 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:26.453743 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:26.454198 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:26.459583 6223 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=45, attempt_no=0}) took 1795 ms. Trace:
I20260921 18:45:26.459661 6223 rpcz_store.cc:276] 0921 18:45:24.663659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:24.663725 (+ 66us) service_pool.cc:227] Handling call
0921 18:45:26.459570 (+1795845us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:24.664051 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:24.664116 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:24.664118 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:24.664119 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:24.664167 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:24.664172 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:24.664174 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:24.664200 (+ 26us) tablet.cc:699] Row locks acquired
0921 18:45:24.664200 (+ 0us) write_op.cc:260] PREPARE: finished
0921 18:45:24.664225 (+ 25us) write_op.cc:270] Start()
0921 18:45:25.643357 (+979132us) write_op.cc:276] Timestamp: P: 1790016324664175 usec, L: 0
0921 18:45:25.643376 (+ 19us) op_driver.cc:331] REPLICATION: starting
0921 18:45:25.643590 (+ 214us) log.cc:842] Serialized 354 byte log entry
0921 18:45:26.458983 (+815393us) op_driver.cc:444] REPLICATION: finished
0921 18:45:26.459179 (+ 196us) write_op.cc:301] APPLY: starting
0921 18:45:26.459213 (+ 34us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:26.459244 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:26.459274 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:26.459331 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:26.459351 (+ 20us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:26.459354 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:26.459401 (+ 47us) log.cc:842] Serialized 22 byte log entry
0921 18:45:26.459484 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:26.459496 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:26.459500 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":134,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":524,"prepare.run_wall_time_us":979594,"replication_time_us":815588,"thread_start_us":61,"threads_started":1}]]}
I20260921 18:45:26.460435 5637 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:48234 (request call id 73) took 1645 ms. Trace:
I20260921 18:45:26.460500 5637 rpcz_store.cc:276] 0921 18:45:24.814566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:24.814625 (+ 59us) service_pool.cc:227] Handling call
0921 18:45:24.814793 (+ 168us) tablet_service.cc:2780] Created scanner f36fa15e14d14fd38f5348244fda54e2 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 2ae1fd5d96f24b12bcd0bf0bc6c42a63
0921 18:45:24.814950 (+ 157us) tablet_service.cc:2920] Creating iterator
0921 18:45:24.814972 (+ 22us) tablet_service.cc:3297] Waiting safe time to advance
0921 18:45:25.643492 (+828520us) tablet_service.cc:3304] Waiting for operations to commit
0921 18:45:26.459514 (+816022us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1644532 microseconds
0921 18:45:26.459590 (+ 76us) tablet_service.cc:2945] Iterator created
0921 18:45:26.459924 (+ 334us) tablet_service.cc:2967] Iterator init: OK
0921 18:45:26.459949 (+ 25us) tablet_service.cc:3010] has_more: true
0921 18:45:26.459977 (+ 28us) tablet_service.cc:3027] Continuing scan request
0921 18:45:26.460038 (+ 61us) tablet_service.cc:3091] Found scanner f36fa15e14d14fd38f5348244fda54e2 for tablet f094ab8ff7d84a9f96e61813d4559ac2, query id is 2ae1fd5d96f24b12bcd0bf0bc6c42a63
0921 18:45:26.460426 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20260921 18:45:26.460964 6030 scanner-internal.cc:458] Time spent opening tablet: real 1.647s user 0.001s sys 0.000s
I20260921 18:45:26.502063 6215 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:26.538621 6223 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=46, attempt_no=0}) took 1472 ms. Trace:
I20260921 18:45:26.538712 6223 rpcz_store.cc:276] 0921 18:45:25.066372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:25.066435 (+ 63us) service_pool.cc:227] Handling call
0921 18:45:26.538604 (+1472169us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:25.643675 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:25.643777 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:25.643784 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:45:25.643785 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:25.643860 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:25.643871 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:25.643874 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:25.643914 (+ 40us) tablet.cc:699] Row locks acquired
0921 18:45:25.643915 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:25.643947 (+ 32us) write_op.cc:270] Start()
0921 18:45:25.697115 (+ 53168us) write_op.cc:276] Timestamp: P: 1790016325643899 usec, L: 0
0921 18:45:25.697119 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:25.697293 (+ 174us) log.cc:842] Serialized 354 byte log entry
0921 18:45:26.538034 (+840741us) op_driver.cc:444] REPLICATION: finished
0921 18:45:26.538143 (+ 109us) write_op.cc:301] APPLY: starting
0921 18:45:26.538174 (+ 31us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:26.538204 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:26.538219 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:26.538294 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:26.538323 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:26.538327 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:26.538401 (+ 74us) log.cc:842] Serialized 22 byte log entry
0921 18:45:26.538485 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:26.538525 (+ 40us) write_op.cc:454] Released schema lock
0921 18:45:26.538530 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":44,"num_ops":1,"prepare.queue_time_us":576943,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":53692,"replication_time_us":840889}]]}
W20260921 18:45:26.540359 6029 scanner-internal.cc:458] Time spent opening tablet: real 0.744s user 0.001s sys 0.000s
I20260921 18:45:26.686283 6194 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:27.368572 6227 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:27.368726 6227 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:27.369374 6227 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:27.369794 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:27.369910 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:27.370296 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:27.370435 6227 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:27.405467 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.794s user 0.001s sys 0.000s
W20260921 18:45:27.405488 6029 scanner-internal.cc:458] Time spent opening tablet: real 0.715s user 0.001s sys 0.000s
I20260921 18:45:27.745995 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:27.747320 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:27.747771 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:27.748144 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:27.748181 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:27.748541 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:27.989843 6227 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:27.990005 6227 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:27.990516 6227 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:27.990896 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:27.991302 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:27.991763 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:27.991940 6227 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:28.040649 6237 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=48, attempt_no=0}) took 1500 ms. Trace:
I20260921 18:45:28.040760 6237 rpcz_store.cc:276] 0921 18:45:26.539871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:26.539911 (+ 40us) service_pool.cc:227] Handling call
0921 18:45:28.040615 (+1500704us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:27.134195 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:27.134317 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:27.134322 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:27.134323 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:27.134401 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:27.134410 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:27.134413 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:27.134454 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:45:27.134455 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:27.134489 (+ 34us) write_op.cc:270] Start()
0921 18:45:27.648630 (+514141us) write_op.cc:276] Timestamp: P: 1790016327134453 usec, L: 0
0921 18:45:27.648633 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:27.648865 (+ 232us) log.cc:842] Serialized 354 byte log entry
0921 18:45:28.039985 (+391120us) op_driver.cc:444] REPLICATION: finished
0921 18:45:28.040269 (+ 284us) write_op.cc:301] APPLY: starting
0921 18:45:28.040283 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:28.040306 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:28.040318 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:28.040364 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:28.040385 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:28.040387 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:28.040443 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:45:28.040529 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:28.040543 (+ 14us) write_op.cc:454] Released schema lock
0921 18:45:28.040549 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":227,"num_ops":1,"prepare.queue_time_us":593907,"prepare.run_cpu_time_us":667,"prepare.run_wall_time_us":514734,"replication_time_us":391326,"thread_start_us":90,"threads_started":1}]]}
I20260921 18:45:28.350551 6194 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:28.769133 6240 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:28.769296 6240 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:28.769816 6240 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:28.770262 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:28.770386 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:28.770985 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:28.771247 6240 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:28.821075 6243 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=49, attempt_no=0}) took 1415 ms. Trace:
I20260921 18:45:28.821226 6243 rpcz_store.cc:276] 0921 18:45:27.405363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:27.405432 (+ 69us) service_pool.cc:227] Handling call
0921 18:45:28.821059 (+1415627us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:27.648960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:27.649063 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:27.649067 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:27.649068 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:27.649141 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:27.649151 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:27.649154 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:27.649198 (+ 44us) tablet.cc:699] Row locks acquired
0921 18:45:27.649199 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:27.649226 (+ 27us) write_op.cc:270] Start()
0921 18:45:28.508337 (+859111us) write_op.cc:276] Timestamp: P: 1790016327649184 usec, L: 0
0921 18:45:28.508339 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:45:28.508562 (+ 223us) log.cc:842] Serialized 354 byte log entry
0921 18:45:28.819358 (+310796us) op_driver.cc:444] REPLICATION: finished
0921 18:45:28.819610 (+ 252us) write_op.cc:301] APPLY: starting
0921 18:45:28.819625 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:28.819650 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:28.819662 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:28.819730 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:28.819753 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:28.819756 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:28.819814 (+ 58us) log.cc:842] Serialized 22 byte log entry
0921 18:45:28.820959 (+ 1145us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:28.820974 (+ 15us) write_op.cc:454] Released schema lock
0921 18:45:28.820979 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":203,"num_ops":1,"prepare.queue_time_us":243160,"prepare.run_cpu_time_us":672,"prepare.run_wall_time_us":859727,"raft.queue_time_us":108,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":310973,"spinlock_wait_cycles":100608,"thread_start_us":115,"threads_started":1}]]}
W20260921 18:45:28.821972 6030 scanner-internal.cc:458] Time spent opening tablet: real 0.630s user 0.000s sys 0.000s
W20260921 18:45:28.822180 6029 scanner-internal.cc:458] Time spent opening tablet: real 0.630s user 0.001s sys 0.000s
I20260921 18:45:29.119379 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:29.119529 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:29.120234 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:29.120599 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:29.120690 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
I20260921 18:45:29.121101 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9819191de3942c1aa42bbfdc863de79; no voters: 2f8374f0de944897bfe1594dd47a2725, 37a20c5ed1254fd09d786fe4099ed76a
I20260921 18:45:29.436866 6240 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:29.437032 6240 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:29.437609 6240 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:29.437959 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:29.437996 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:29.438398 5742 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:29.438553 6240 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:29.482523 6249 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48234 (ReqId={client: 7bca6c4463334921998ef379a33eaae0, seq_no=50, attempt_no=0}) took 1440 ms. Trace:
I20260921 18:45:29.482640 6249 rpcz_store.cc:276] 0921 18:45:28.042319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:28.042393 (+ 74us) service_pool.cc:227] Handling call
0921 18:45:29.482504 (+1440111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:28.508710 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f094ab8ff7d84a9f96e61813d4559ac2
0921 18:45:28.508812 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:28.508815 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:28.508817 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:28.508897 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:28.508904 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:28.508907 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:28.508945 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:45:28.508947 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:28.508983 (+ 36us) write_op.cc:270] Start()
0921 18:45:28.935134 (+426151us) write_op.cc:276] Timestamp: P: 1790016328508958 usec, L: 0
0921 18:45:28.935137 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:28.935321 (+ 184us) log.cc:842] Serialized 354 byte log entry
0921 18:45:29.481950 (+546629us) op_driver.cc:444] REPLICATION: finished
0921 18:45:29.482169 (+ 219us) write_op.cc:301] APPLY: starting
0921 18:45:29.482183 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:29.482208 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:29.482220 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:29.482267 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:29.482289 (+ 22us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:29.482291 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:29.482348 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:45:29.482415 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:29.482429 (+ 14us) write_op.cc:454] Released schema lock
0921 18:45:29.482434 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":172,"num_ops":1,"prepare.queue_time_us":466052,"prepare.run_cpu_time_us":598,"prepare.run_wall_time_us":426671,"replication_time_us":546789,"thread_start_us":76,"threads_started":1}]]}
I20260921 18:45:29.546447 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:45:29.547564 5658 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:45:29.561928 6240 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:29.562047 6240 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:29.562412 6240 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:29.562767 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:29.563262 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:29.563587 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:29.563818 6240 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:29.715473 6240 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:29.715602 6240 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:29.715999 6240 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:29.716588 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:29.717615 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "37a20c5ed1254fd09d786fe4099ed76a" is_pre_election: true
I20260921 18:45:29.718093 5745 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725; no voters: 37a20c5ed1254fd09d786fe4099ed76a, b9819191de3942c1aa42bbfdc863de79
I20260921 18:45:29.718312 6240 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:30.103245 6194 raft_consensus.cc:2726] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:30.105393 5920 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
5b4c8580482948e19362fe2e4df6ba0c | 127.3.36.190:33165 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:39217 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2f8374f0de944897bfe1594dd47a2725 | 127.3.36.130:38815 | HEALTHY | <none> | 0 | 0
37a20c5ed1254fd09d786fe4099ed76a | 127.3.36.129:38133 | HEALTHY | <none> | 1 | 0
b9819191de3942c1aa42bbfdc863de79 | 127.3.36.131:41413 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:38133
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:38815
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:41413
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:38133
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:38815
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:41413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:39217 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260921 18:45:30.129426 3218 log_verifier.cc:126] Checking tablet f094ab8ff7d84a9f96e61813d4559ac2
I20260921 18:45:30.173066 3218 log_verifier.cc:177] Verified matching terms for 52 ops in tablet f094ab8ff7d84a9f96e61813d4559ac2
I20260921 18:45:30.193231 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 5596
I20260921 18:45:30.199229 6240 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:30.199352 6240 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:30.199445 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:30.199558 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:30.199748 6240 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), b9819191de3942c1aa42bbfdc863de79 (127.3.36.131:41413)
I20260921 18:45:30.199967 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
I20260921 18:45:30.200121 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "2f8374f0de944897bfe1594dd47a2725" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "b9819191de3942c1aa42bbfdc863de79" is_pre_election: true
I20260921 18:45:30.200243 5940 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f8374f0de944897bfe1594dd47a2725 in term 1.
W20260921 18:45:30.200428 5876 connection.cc:535] client connection to 127.3.36.129:38133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:45:30.200445 5809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f094ab8ff7d84a9f96e61813d4559ac2" candidate_uuid: "b9819191de3942c1aa42bbfdc863de79" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "2f8374f0de944897bfe1594dd47a2725" is_pre_election: true
W20260921 18:45:30.200534 5876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20260921 18:45:30.200559 5809 raft_consensus.cc:2454] T f094ab8ff7d84a9f96e61813d4559ac2 P 2f8374f0de944897bfe1594dd47a2725 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9819191de3942c1aa42bbfdc863de79 in term 1.
I20260921 18:45:30.200762 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 5727
W20260921 18:45:30.200788 5745 connection.cc:535] client connection to 127.3.36.129:38133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:45:30.200829 5874 leader_election.cc:302] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f8374f0de944897bfe1594dd47a2725, b9819191de3942c1aa42bbfdc863de79; no voters:
I20260921 18:45:30.202020 6194 raft_consensus.cc:2781] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260921 18:45:30.202082 6194 raft_consensus.cc:482] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37a20c5ed1254fd09d786fe4099ed76a)
I20260921 18:45:30.202137 6194 raft_consensus.cc:3037] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 1 FOLLOWER]: Advancing to term 2
W20260921 18:45:30.202781 5876 leader_election.cc:334] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133): Network error: Client connection negotiation failed: client connection to 127.3.36.129:38133: connect: Connection refused (error 111)
I20260921 18:45:30.203348 6194 raft_consensus.cc:504] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37a20c5ed1254fd09d786fe4099ed76a" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38133 } } peers { permanent_uuid: "2f8374f0de944897bfe1594dd47a2725" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 38815 } } peers { permanent_uuid: "b9819191de3942c1aa42bbfdc863de79" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 41413 } }
I20260921 18:45:30.203708 6194 leader_election.cc:288] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 election: Requested vote from peers 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133), 2f8374f0de944897bfe1594dd47a2725 (127.3.36.130:38815)
W20260921 18:45:30.204566 5876 leader_election.cc:334] T f094ab8ff7d84a9f96e61813d4559ac2 P b9819191de3942c1aa42bbfdc863de79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37a20c5ed1254fd09d786fe4099ed76a (127.3.36.129:38133): Network error: Client connection negotiation failed: client connection to 127.3.36.129:38133: connect: Connection refused (error 111)
W20260921 18:45:30.206811 5875 connection.cc:535] server connection from 127.3.36.130:34919 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260921 18:45:30.206858 5874 connection.cc:535] client connection to 127.3.36.130:38815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:45:30.207345 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 5858
I20260921 18:45:30.212709 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 5505
2026-09-21T18:45:30Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (32416 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20260921 18:45:30.236275 3218 test_util.cc:268] Using random seed: -1484684946
I20260921 18:45:30.237010 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:45:30.237061 3218 ts_itest-base.cc:113] --------------
I20260921 18:45:30.237072 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:45:30.237083 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:45:30.237093 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:45:30Z Disabled control of system clock
I20260921 18:45:30.245292 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:46581
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:46581 with env {}
W20260921 18:45:30.349642 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:45:30.349915 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:45:30.350003 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:45:30.353987 6298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:45:30.354063 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:45:30.354096 6298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:45:30.354151 6298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:45:30.358742 6298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:46581
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:45:30.359714 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:45:30.361032 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260921 18:45:30.369155 6298 server_base.cc:1034] running on GCE node
W20260921 18:45:30.369109 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.369135 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.369109 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:45:30.369863 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:45:30.370769 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:45:30.371922 6298 hybrid_clock.cc:648] HybridClock initialized: now 1790016330371891 us; error 40 us; skew 500 ppm
I20260921 18:45:30.373600 6298 webserver.cc:460] Webserver started at http://127.3.36.190:36461/ using document root <none> and password file <none>
I20260921 18:45:30.374143 6298 fs_manager.cc:359] Metadata directory not provided
I20260921 18:45:30.374200 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:45:30.374423 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:45:30.376147 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "aff9f6524d084d6e8e8151b0b2225e35"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.376670 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "aff9f6524d084d6e8e8151b0b2225e35"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.380059 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260921 18:45:30.382233 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.383359 6298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:45:30.383487 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "aff9f6524d084d6e8e8151b0b2225e35"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.383585 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:45:30.406957 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:45:30.407574 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:45:30.407759 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:45:30.414204 6298 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:46581
I20260921 18:45:30.414215 6364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:46581 every 8 connection(s)
I20260921 18:45:30.415071 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:45:30.418028 6365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:45:30.420137 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 6298
I20260921 18:45:30.420265 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:45:30.422951 6365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35: Bootstrap starting.
I20260921 18:45:30.425323 6365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35: Neither blocks nor log segments found. Creating new log.
I20260921 18:45:30.426162 6365 log.cc:824] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35: Log is configured to *not* fsync() on all Append() calls
I20260921 18:45:30.428052 6365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35: No bootstrap required, opened a new log
I20260921 18:45:30.431787 6365 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9f6524d084d6e8e8151b0b2225e35" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46581 } }
I20260921 18:45:30.432039 6365 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:45:30.432099 6365 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff9f6524d084d6e8e8151b0b2225e35, State: Initialized, Role: FOLLOWER
I20260921 18:45:30.432525 6365 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9f6524d084d6e8e8151b0b2225e35" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46581 } }
I20260921 18:45:30.432660 6365 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:45:30.432720 6365 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:45:30.432812 6365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:45:30.433564 6365 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9f6524d084d6e8e8151b0b2225e35" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46581 } }
I20260921 18:45:30.433868 6365 leader_election.cc:302] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff9f6524d084d6e8e8151b0b2225e35; no voters:
I20260921 18:45:30.434183 6365 leader_election.cc:288] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:45:30.434265 6370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:45:30.434448 6370 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [term 1 LEADER]: Becoming Leader. State: Replica: aff9f6524d084d6e8e8151b0b2225e35, State: Running, Role: LEADER
I20260921 18:45:30.434767 6370 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9f6524d084d6e8e8151b0b2225e35" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46581 } }
I20260921 18:45:30.435316 6365 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:45:30.436456 6371 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff9f6524d084d6e8e8151b0b2225e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9f6524d084d6e8e8151b0b2225e35" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46581 } } }
I20260921 18:45:30.436484 6372 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff9f6524d084d6e8e8151b0b2225e35. Latest consensus state: current_term: 1 leader_uuid: "aff9f6524d084d6e8e8151b0b2225e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff9f6524d084d6e8e8151b0b2225e35" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46581 } } }
I20260921 18:45:30.436561 6371 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [sys.catalog]: This master's current role is: LEADER
I20260921 18:45:30.436578 6372 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35 [sys.catalog]: This master's current role is: LEADER
I20260921 18:45:30.436957 6379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:45:30.439637 6379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:45:30.444727 6379 catalog_manager.cc:1347] Generated new cluster ID: cd8b6db1f69447c6a99deea8ebe9bbd4
I20260921 18:45:30.444799 6379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:45:30.449821 6379 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:45:30.450882 6379 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:45:30.458256 6379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff9f6524d084d6e8e8151b0b2225e35: Generated new TSK 0
I20260921 18:45:30.458856 6379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:45:30.461879 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:46581
--builtin_ntp_servers=127.3.36.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260921 18:45:30.578258 6389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:45:30.578488 6389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:45:30.578524 6389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:45:30.582541 6389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:45:30.582629 6389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:45:30.587277 6389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:46581
--never_fsync=true
--heap_profile_path=/tmp/kudu.6389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:45:30.588296 6389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:45:30.589691 6389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:45:30.598438 6394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.598452 6395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.598452 6397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:45:30.598637 6389 server_base.cc:1034] running on GCE node
I20260921 18:45:30.599033 6389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:45:30.599610 6389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:45:30.600785 6389 hybrid_clock.cc:648] HybridClock initialized: now 1790016330600748 us; error 48 us; skew 500 ppm
I20260921 18:45:30.602499 6389 webserver.cc:460] Webserver started at http://127.3.36.129:41965/ using document root <none> and password file <none>
I20260921 18:45:30.603066 6389 fs_manager.cc:359] Metadata directory not provided
I20260921 18:45:30.603154 6389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:45:30.603375 6389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:45:30.605111 6389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3ed466b1d5d14005adcb29914704ab7c"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.605751 6389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3ed466b1d5d14005adcb29914704ab7c"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.609097 6389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260921 18:45:30.611384 6403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.612448 6389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:45:30.612578 6389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3ed466b1d5d14005adcb29914704ab7c"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.612681 6389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:45:30.625504 6389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:45:30.626142 6389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:45:30.626328 6389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:45:30.626866 6389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:45:30.627738 6389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:45:30.627825 6389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.627910 6389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:45:30.627962 6389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.636397 6389 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:38457
I20260921 18:45:30.636452 6516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:38457 every 8 connection(s)
I20260921 18:45:30.637391 6389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:45:30.638515 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 6389
I20260921 18:45:30.638641 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:45:30.641067 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:46581
--builtin_ntp_servers=127.3.36.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20260921 18:45:30.648465 6517 heartbeater.cc:347] Connected to a master server at 127.3.36.190:46581
I20260921 18:45:30.648720 6517 heartbeater.cc:464] Registering TS with master...
I20260921 18:45:30.649322 6517 heartbeater.cc:511] Master 127.3.36.190:46581 requested a full tablet report, sending...
I20260921 18:45:30.650805 6329 ts_manager.cc:194] Registered new tserver with Master: 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457)
I20260921 18:45:30.651897 6329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:37835
W20260921 18:45:30.748637 6520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:45:30.748876 6520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:45:30.748950 6520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:45:30.752930 6520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:45:30.753058 6520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:45:30.757748 6520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:46581
--never_fsync=true
--heap_profile_path=/tmp/kudu.6520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:45:30.758776 6520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:45:30.760097 6520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:45:30.769163 6525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.769201 6528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.769189 6526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:45:30.769310 6520 server_base.cc:1034] running on GCE node
I20260921 18:45:30.769800 6520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:45:30.770360 6520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:45:30.771512 6520 hybrid_clock.cc:648] HybridClock initialized: now 1790016330771503 us; error 28 us; skew 500 ppm
I20260921 18:45:30.773151 6520 webserver.cc:460] Webserver started at http://127.3.36.130:33007/ using document root <none> and password file <none>
I20260921 18:45:30.773713 6520 fs_manager.cc:359] Metadata directory not provided
I20260921 18:45:30.773773 6520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:45:30.773927 6520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:45:30.775635 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7a46f3a6b1634626be69e0782c521f15"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.776109 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7a46f3a6b1634626be69e0782c521f15"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.779618 6520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260921 18:45:30.781823 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.783005 6520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:45:30.783126 6520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7a46f3a6b1634626be69e0782c521f15"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.783224 6520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:45:30.805755 6520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:45:30.806414 6520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:45:30.806604 6520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:45:30.807157 6520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:45:30.808046 6520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:45:30.808117 6520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.808187 6520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:45:30.808226 6520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.816489 6520 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:36893
I20260921 18:45:30.816531 6647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:36893 every 8 connection(s)
I20260921 18:45:30.817327 6520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:45:30.826614 6648 heartbeater.cc:347] Connected to a master server at 127.3.36.190:46581
I20260921 18:45:30.826843 6648 heartbeater.cc:464] Registering TS with master...
I20260921 18:45:30.827173 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 6520
I20260921 18:45:30.827294 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:45:30.827346 6648 heartbeater.cc:511] Master 127.3.36.190:46581 requested a full tablet report, sending...
I20260921 18:45:30.828395 6328 ts_manager.cc:194] Registered new tserver with Master: 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:30.829072 6328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:52651
I20260921 18:45:30.829380 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:46581
--builtin_ntp_servers=127.3.36.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20260921 18:45:30.933851 6651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:45:30.934108 6651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:45:30.934173 6651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:45:30.938587 6651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:45:30.938714 6651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:45:30.943207 6651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.3.36.190:46581
--never_fsync=true
--heap_profile_path=/tmp/kudu.6651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:45:30.944121 6651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:45:30.945389 6651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:45:30.954228 6659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.954238 6656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:45:30.954276 6657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:45:30.954546 6651 server_base.cc:1034] running on GCE node
I20260921 18:45:30.954901 6651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:45:30.955451 6651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:45:30.956609 6651 hybrid_clock.cc:648] HybridClock initialized: now 1790016330956595 us; error 26 us; skew 500 ppm
I20260921 18:45:30.958438 6651 webserver.cc:460] Webserver started at http://127.3.36.131:35295/ using document root <none> and password file <none>
I20260921 18:45:30.958992 6651 fs_manager.cc:359] Metadata directory not provided
I20260921 18:45:30.959050 6651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:45:30.959272 6651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:45:30.960989 6651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dd3f9f8b70dc432699d59db2668757d9"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.961565 6651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dd3f9f8b70dc432699d59db2668757d9"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.964876 6651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260921 18:45:30.967164 6665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.968245 6651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260921 18:45:30.968374 6651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dd3f9f8b70dc432699d59db2668757d9"
format_stamp: "Formatted at 2026-09-21 18:45:30 on dist-test-slave-l4st"
I20260921 18:45:30.968474 6651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:45:30.984038 6651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:45:30.984671 6651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:45:30.984844 6651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:45:30.985359 6651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:45:30.986348 6651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:45:30.986416 6651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.986476 6651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:45:30.986506 6651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:45:30.995316 6651 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:36181
I20260921 18:45:30.995329 6778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:36181 every 8 connection(s)
I20260921 18:45:30.996148 6651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:45:31.005054 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 6651
I20260921 18:45:31.005172 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:45:31.005869 6779 heartbeater.cc:347] Connected to a master server at 127.3.36.190:46581
I20260921 18:45:31.006105 6779 heartbeater.cc:464] Registering TS with master...
I20260921 18:45:31.006702 6779 heartbeater.cc:511] Master 127.3.36.190:46581 requested a full tablet report, sending...
I20260921 18:45:31.007692 6328 ts_manager.cc:194] Registered new tserver with Master: dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:31.008257 6328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:42431
I20260921 18:45:31.018741 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:45:31.020079 3218 test_util.cc:268] Using random seed: -1483901140
I20260921 18:45:31.030246 6328 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40212:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260921 18:45:31.031226 6328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:45:31.047185 6451 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ae6eac23448b79a40085d1e15c4d3 (DEFAULT_TABLE table=test-workload [id=6fcd5c6e5eaf4ad5983b40590de01d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:45:31.047539 6582 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ae6eac23448b79a40085d1e15c4d3 (DEFAULT_TABLE table=test-workload [id=6fcd5c6e5eaf4ad5983b40590de01d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:45:31.047533 6713 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4ae6eac23448b79a40085d1e15c4d3 (DEFAULT_TABLE table=test-workload [id=6fcd5c6e5eaf4ad5983b40590de01d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:45:31.048128 6451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ae6eac23448b79a40085d1e15c4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:45:31.048518 6713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ae6eac23448b79a40085d1e15c4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:45:31.048528 6582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4ae6eac23448b79a40085d1e15c4d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:45:31.053396 6796 tablet_bootstrap.cc:492] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c: Bootstrap starting.
I20260921 18:45:31.054725 6797 tablet_bootstrap.cc:492] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9: Bootstrap starting.
I20260921 18:45:31.056542 6796 tablet_bootstrap.cc:654] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c: Neither blocks nor log segments found. Creating new log.
I20260921 18:45:31.056839 6797 tablet_bootstrap.cc:654] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9: Neither blocks nor log segments found. Creating new log.
I20260921 18:45:31.057399 6796 log.cc:824] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c: Log is configured to *not* fsync() on all Append() calls
I20260921 18:45:31.057538 6797 log.cc:824] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9: Log is configured to *not* fsync() on all Append() calls
I20260921 18:45:31.058054 6798 tablet_bootstrap.cc:492] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: Bootstrap starting.
I20260921 18:45:31.059317 6796 tablet_bootstrap.cc:492] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c: No bootstrap required, opened a new log
I20260921 18:45:31.059497 6796 ts_tablet_manager.cc:1397] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260921 18:45:31.060027 6798 tablet_bootstrap.cc:654] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: Neither blocks nor log segments found. Creating new log.
I20260921 18:45:31.060772 6798 log.cc:824] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: Log is configured to *not* fsync() on all Append() calls
I20260921 18:45:31.062295 6798 tablet_bootstrap.cc:492] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: No bootstrap required, opened a new log
I20260921 18:45:31.062372 6797 tablet_bootstrap.cc:492] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9: No bootstrap required, opened a new log
I20260921 18:45:31.062484 6798 ts_tablet_manager.cc:1397] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:45:31.062543 6797 ts_tablet_manager.cc:1397] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260921 18:45:31.063810 6796 raft_consensus.cc:348] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.064090 6796 raft_consensus.cc:374] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:45:31.064177 6796 raft_consensus.cc:729] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed466b1d5d14005adcb29914704ab7c, State: Initialized, Role: FOLLOWER
I20260921 18:45:31.064634 6796 consensus_queue.cc:260] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.065256 6517 heartbeater.cc:503] Master 127.3.36.190:46581 was elected leader, sending a full tablet report...
I20260921 18:45:31.065673 6802 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:45:31.065863 6802 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.066082 6798 raft_consensus.cc:348] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.066327 6798 raft_consensus.cc:374] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:45:31.066387 6798 raft_consensus.cc:729] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a46f3a6b1634626be69e0782c521f15, State: Initialized, Role: FOLLOWER
I20260921 18:45:31.066581 6797 raft_consensus.cc:348] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.066795 6797 raft_consensus.cc:374] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:45:31.066854 6797 raft_consensus.cc:729] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3f9f8b70dc432699d59db2668757d9, State: Initialized, Role: FOLLOWER
I20260921 18:45:31.066836 6798 consensus_queue.cc:260] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.067283 6797 consensus_queue.cc:260] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.067768 6798 ts_tablet_manager.cc:1428] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260921 18:45:31.068104 6649 tablet.cc:2369] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:45:31.068460 6797 ts_tablet_manager.cc:1428] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260921 18:45:31.068749 6779 heartbeater.cc:503] Master 127.3.36.190:46581 was elected leader, sending a full tablet report...
I20260921 18:45:31.068985 6648 heartbeater.cc:503] Master 127.3.36.190:46581 was elected leader, sending a full tablet report...
I20260921 18:45:31.069725 6806 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:45:31.069837 6806 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.070799 6802 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:31.071728 6806 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:31.071992 6803 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:45:31.072115 6803 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.073242 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:31.073419 6733 raft_consensus.cc:2454] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed466b1d5d14005adcb29914704ab7c in term 0.
I20260921 18:45:31.073863 6405 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9; no voters:
I20260921 18:45:31.074117 6802 raft_consensus.cc:2781] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:45:31.074189 6802 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:45:31.074231 6802 raft_consensus.cc:3037] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:45:31.074743 6803 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
W20260921 18:45:31.075228 6404 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.3.36.130:36893 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20260921 18:45:31.075487 6802 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.076215 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9"
I20260921 18:45:31.076327 6733 raft_consensus.cc:3037] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:45:31.077225 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:31.077462 6471 raft_consensus.cc:2379] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3f9f8b70dc432699d59db2668757d9 in current term 1: Already voted for candidate 3ed466b1d5d14005adcb29914704ab7c in this term.
I20260921 18:45:31.077497 6733 raft_consensus.cc:2454] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed466b1d5d14005adcb29914704ab7c in term 1.
W20260921 18:45:31.077776 6667 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.36.129:38457 timed out after 0.006s (SENT)
I20260921 18:45:31.077986 6802 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 1 election: Requested vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:31.078142 6405 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9; no voters:
I20260921 18:45:31.078353 6802 raft_consensus.cc:2781] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:45:31.078526 6802 raft_consensus.cc:686] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 LEADER]: Becoming Leader. State: Replica: 3ed466b1d5d14005adcb29914704ab7c, State: Running, Role: LEADER
I20260921 18:45:31.078809 6802 consensus_queue.cc:237] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.079052 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15"
I20260921 18:45:31.079192 6602 raft_consensus.cc:3037] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:45:31.080334 6602 raft_consensus.cc:2454] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed466b1d5d14005adcb29914704ab7c in term 1.
I20260921 18:45:31.080711 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
W20260921 18:45:31.081073 6536 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.36.129:38457 timed out after 0.007s (SENT)
I20260921 18:45:31.081256 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:31.081374 6602 raft_consensus.cc:2362] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3ed466b1d5d14005adcb29914704ab7c in term 1. Re-sending same reply.
I20260921 18:45:31.082278 6796 ts_tablet_manager.cc:1428] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20260921 18:45:31.082614 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:31.082898 6328 catalog_manager.cc:5495] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129). New cstate: current_term: 1 leader_uuid: "3ed466b1d5d14005adcb29914704ab7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260921 18:45:31.084385 6802 consensus_queue.cc:1035] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:45:31.084483 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:31.084520 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:31.084640 6602 raft_consensus.cc:2379] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3f9f8b70dc432699d59db2668757d9 in current term 1: Already voted for candidate 3ed466b1d5d14005adcb29914704ab7c in this term.
I20260921 18:45:31.085009 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:31.085212 6814 consensus_queue.cc:1035] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:45:31.085527 6805 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260921 18:45:31.085690 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:31.085824 6733 raft_consensus.cc:2379] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a46f3a6b1634626be69e0782c521f15 in current term 1: Already voted for candidate 3ed466b1d5d14005adcb29914704ab7c in this term.
I20260921 18:45:31.086258 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:31.086771 6803 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260921 18:45:31.101670 6823 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:45:31.101765 6823 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260921 18:45:31.648579 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:31.648736 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:31.649365 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:31.649425 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:31.649518 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:31.649863 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:31.650121 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260921 18:45:31.816008 6733 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:45:31.816022 6602 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:45:31.816088 6602 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20260921 18:45:31.816088 6733 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20260921 18:45:32.406781 6828 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:32.406955 6828 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:32.408248 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:32.408622 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:32.408955 6828 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:32.409061 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:32.409263 6828 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:32.410012 6833 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=0, attempt_no=0}) took 1312 ms. Trace:
I20260921 18:45:32.410118 6833 rpcz_store.cc:276] 0921 18:45:31.097589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:31.097648 (+ 59us) service_pool.cc:227] Handling call
0921 18:45:32.410002 (+1312354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:31.099958 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:31.100080 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:31.100095 (+ 15us) write_op.cc:435] Acquired schema lock
0921 18:45:31.100098 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:31.100585 (+ 487us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:31.100650 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:31.100655 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:31.101462 (+ 807us) tablet.cc:699] Row locks acquired
0921 18:45:31.101463 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:31.101601 (+ 138us) write_op.cc:270] Start()
0921 18:45:31.814161 (+712560us) write_op.cc:276] Timestamp: P: 1790016331101517 usec, L: 0
0921 18:45:31.814166 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:31.814529 (+ 363us) log.cc:842] Serialized 345 byte log entry
0921 18:45:32.408111 (+593582us) op_driver.cc:444] REPLICATION: finished
0921 18:45:32.408346 (+ 235us) write_op.cc:301] APPLY: starting
0921 18:45:32.408407 (+ 61us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:32.408642 (+ 235us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:32.408670 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:32.408990 (+ 320us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:32.409155 (+ 165us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:32.409178 (+ 23us) write_op.cc:312] APPLY: finished
0921 18:45:32.409424 (+ 246us) log.cc:842] Serialized 22 byte log entry
0921 18:45:32.409748 (+ 324us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:32.409793 (+ 45us) write_op.cc:454] Released schema lock
0921 18:45:32.409801 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":384,"child_traces":[["op",{"apply.queue_time_us":176,"num_ops":1,"prepare.queue_time_us":73,"prepare.run_cpu_time_us":2618,"prepare.run_wall_time_us":714748,"replication_time_us":593894,"thread_start_us":67,"threads_started":1}]]}
I20260921 18:45:32.650527 6833 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=1, attempt_no=0}) took 1552 ms. Trace:
I20260921 18:45:32.650625 6833 rpcz_store.cc:276] 0921 18:45:31.097744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:31.097803 (+ 59us) service_pool.cc:227] Handling call
0921 18:45:32.650511 (+1552708us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:31.814652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:31.814774 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:31.814781 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:45:31.814784 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:31.814859 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:31.814894 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:31.814897 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:31.814940 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:45:31.814943 (+ 3us) write_op.cc:260] PREPARE: finished
0921 18:45:31.815000 (+ 57us) write_op.cc:270] Start()
0921 18:45:32.644125 (+829125us) write_op.cc:276] Timestamp: P: 1790016331814957 usec, L: 0
0921 18:45:32.644128 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:32.644271 (+ 143us) log.cc:842] Serialized 345 byte log entry
0921 18:45:32.650018 (+ 5747us) op_driver.cc:444] REPLICATION: finished
0921 18:45:32.650131 (+ 113us) write_op.cc:301] APPLY: starting
0921 18:45:32.650146 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:32.650193 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:32.650209 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:32.650282 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:32.650317 (+ 35us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:32.650320 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:32.650389 (+ 69us) log.cc:842] Serialized 22 byte log entry
0921 18:45:32.650439 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:32.650448 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:32.650452 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1470848,"child_traces":[["op",{"apply.queue_time_us":50,"mutex_wait_us":33,"num_ops":1,"prepare.queue_time_us":714863,"prepare.run_cpu_time_us":641,"prepare.run_wall_time_us":829710,"raft.queue_time_us":42,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":117,"replication_time_us":5872,"spinlock_wait_cycles":30976}]]}
I20260921 18:45:32.792832 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:32.792951 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:32.793651 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:32.794085 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:32.794144 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:32.794708 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:33.011169 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:33.878410 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:33.878548 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:33.879120 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:33.879521 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:33.879575 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:33.880071 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:33.880505 6842 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=2, attempt_no=0}) took 1468 ms. Trace:
I20260921 18:45:33.880578 6842 rpcz_store.cc:276] 0921 18:45:32.411981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:32.412067 (+ 86us) service_pool.cc:227] Handling call
0921 18:45:33.880491 (+1468424us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:32.644393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:32.644505 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:32.644509 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:32.644511 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:32.644584 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:32.644596 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:32.644601 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:32.644643 (+ 42us) tablet.cc:699] Row locks acquired
0921 18:45:32.644649 (+ 6us) write_op.cc:260] PREPARE: finished
0921 18:45:32.644678 (+ 29us) write_op.cc:270] Start()
0921 18:45:33.337827 (+693149us) write_op.cc:276] Timestamp: P: 1790016332644664 usec, L: 0
0921 18:45:33.337831 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:33.338031 (+ 200us) log.cc:842] Serialized 354 byte log entry
0921 18:45:33.879619 (+541588us) op_driver.cc:444] REPLICATION: finished
0921 18:45:33.879958 (+ 339us) write_op.cc:301] APPLY: starting
0921 18:45:33.879976 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:33.880004 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:33.880020 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:33.880071 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:33.880099 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:33.880103 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:33.880183 (+ 80us) log.cc:842] Serialized 22 byte log entry
0921 18:45:33.880400 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:33.880411 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:33.880416 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":288,"num_ops":1,"prepare.queue_time_us":231913,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":693829,"raft.queue_time_us":130,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":541766,"spinlock_wait_cycles":166144,"thread_start_us":62,"threads_started":1}]]}
I20260921 18:45:34.190866 6838 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:34.191000 6838 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:34.191537 6838 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:34.191920 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:34.191955 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:34.192391 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:34.319208 6838 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:34.320652 6842 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=3, attempt_no=0}) took 1668 ms. Trace:
I20260921 18:45:34.320777 6842 rpcz_store.cc:276] 0921 18:45:32.652002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:32.652072 (+ 70us) service_pool.cc:227] Handling call
0921 18:45:34.320633 (+1668561us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:33.338248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:33.338344 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:33.338348 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:33.338349 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:33.338429 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:33.338437 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:33.338439 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:33.338505 (+ 66us) tablet.cc:699] Row locks acquired
0921 18:45:33.338506 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:33.338546 (+ 40us) write_op.cc:270] Start()
0921 18:45:33.456681 (+118135us) write_op.cc:276] Timestamp: P: 1790016333338510 usec, L: 0
0921 18:45:33.456684 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:33.456866 (+ 182us) log.cc:842] Serialized 354 byte log entry
0921 18:45:34.320081 (+863215us) op_driver.cc:444] REPLICATION: finished
0921 18:45:34.320205 (+ 124us) write_op.cc:301] APPLY: starting
0921 18:45:34.320229 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:34.320275 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:34.320294 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:34.320347 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:34.320378 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:34.320386 (+ 8us) write_op.cc:312] APPLY: finished
0921 18:45:34.320472 (+ 86us) log.cc:842] Serialized 22 byte log entry
0921 18:45:34.320554 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:34.320564 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:34.320569 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"num_ops":1,"prepare.queue_time_us":685907,"prepare.run_cpu_time_us":611,"prepare.run_wall_time_us":118675,"replication_time_us":863374}]]}
I20260921 18:45:34.363958 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:34.591589 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:34.591741 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:34.592259 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:34.592612 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:34.592734 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:34.593210 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:35.228392 6838 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:35.228535 6838 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:35.229043 6838 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:35.229490 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:35.229631 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:35.230130 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:35.462872 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:35.464416 6852 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=5, attempt_no=0}) took 1142 ms. Trace:
I20260921 18:45:35.464596 6852 rpcz_store.cc:276] 0921 18:45:34.322130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:34.322178 (+ 48us) service_pool.cc:227] Handling call
0921 18:45:35.464400 (+1142222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:34.361851 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:34.361947 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:34.361953 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:45:34.361956 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:34.362028 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:34.362035 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:34.362036 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:34.362069 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:45:34.362070 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:34.362101 (+ 31us) write_op.cc:270] Start()
0921 18:45:34.483253 (+121152us) write_op.cc:276] Timestamp: P: 1790016334362063 usec, L: 0
0921 18:45:34.483257 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:34.483451 (+ 194us) log.cc:842] Serialized 354 byte log entry
0921 18:45:35.463818 (+980367us) op_driver.cc:444] REPLICATION: finished
0921 18:45:35.464011 (+ 193us) write_op.cc:301] APPLY: starting
0921 18:45:35.464028 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:35.464059 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:35.464098 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:35.464158 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:35.464182 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:35.464185 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:35.464242 (+ 57us) log.cc:842] Serialized 22 byte log entry
0921 18:45:35.464311 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:35.464321 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:35.464326 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141,"num_ops":1,"prepare.queue_time_us":39313,"prepare.run_cpu_time_us":610,"prepare.run_wall_time_us":121675,"replication_time_us":980540,"thread_start_us":63,"threads_started":1}]]}
I20260921 18:45:35.618316 6838 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:35.619577 6852 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=6, attempt_no=0}) took 1025 ms. Trace:
I20260921 18:45:35.619673 6852 rpcz_store.cc:276] 0921 18:45:34.594053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:34.594139 (+ 86us) service_pool.cc:227] Handling call
0921 18:45:35.619565 (+1025426us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:34.594528 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:34.594613 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:34.594616 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:34.594617 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:34.594666 (+ 49us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:34.594671 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:34.594673 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:34.594704 (+ 31us) tablet.cc:699] Row locks acquired
0921 18:45:34.594705 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:34.594727 (+ 22us) write_op.cc:270] Start()
0921 18:45:34.660856 (+ 66129us) write_op.cc:276] Timestamp: P: 1790016334594669 usec, L: 0
0921 18:45:34.660858 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:45:34.661026 (+ 168us) log.cc:842] Serialized 354 byte log entry
0921 18:45:35.619076 (+958050us) op_driver.cc:444] REPLICATION: finished
0921 18:45:35.619167 (+ 91us) write_op.cc:301] APPLY: starting
0921 18:45:35.619181 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:35.619204 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:35.619219 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:35.619271 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:35.619330 (+ 59us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:35.619334 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:35.619394 (+ 60us) log.cc:842] Serialized 22 byte log entry
0921 18:45:35.619470 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:35.619483 (+ 13us) write_op.cc:454] Released schema lock
0921 18:45:35.619488 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":484,"prepare.run_wall_time_us":66549,"replication_time_us":958195}]]}
I20260921 18:45:36.235445 6838 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:36.235625 6838 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:36.236088 6838 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:36.236496 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:36.236544 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:36.237116 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:36.237321 6838 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:36.314651 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:36.314805 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:36.315266 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:36.315677 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:36.315688 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:36.316064 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:36.316260 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:36.928529 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:36.928675 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:36.929196 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:36.929596 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:36.929729 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:36.930047 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:37.209759 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:37.211865 6866 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=8, attempt_no=0}) took 1590 ms. Trace:
I20260921 18:45:37.211954 6866 rpcz_store.cc:276] 0921 18:45:35.621029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:35.621086 (+ 57us) service_pool.cc:227] Handling call
0921 18:45:37.211851 (+1590765us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:35.805954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:35.806059 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:35.806063 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:35.806065 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:35.806156 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:35.806164 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:35.806166 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:35.806199 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:45:35.806200 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:35.806232 (+ 32us) write_op.cc:270] Start()
0921 18:45:36.742369 (+936137us) write_op.cc:276] Timestamp: P: 1790016335806206 usec, L: 0
0921 18:45:36.742372 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:36.742531 (+ 159us) log.cc:842] Serialized 354 byte log entry
0921 18:45:37.210648 (+468117us) op_driver.cc:444] REPLICATION: finished
0921 18:45:37.210864 (+ 216us) write_op.cc:301] APPLY: starting
0921 18:45:37.210878 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:37.210905 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:37.210922 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:37.211039 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:37.211060 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:37.211063 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:37.211185 (+ 122us) log.cc:842] Serialized 43 byte log entry
0921 18:45:37.211258 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:37.211268 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:37.211273 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"num_ops":1,"prepare.queue_time_us":184521,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":936648,"replication_time_us":468253,"thread_start_us":78,"threads_started":1}]]}
I20260921 18:45:37.479664 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:37.479825 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:37.480392 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:37.480778 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:37.480814 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:37.481216 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:38.088135 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:38.089886 6874 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=9, attempt_no=0}) took 1850 ms. Trace:
I20260921 18:45:38.089991 6874 rpcz_store.cc:276] 0921 18:45:36.238946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:36.238999 (+ 53us) service_pool.cc:227] Handling call
0921 18:45:38.089868 (+1850869us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:36.742642 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:36.742745 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:36.742749 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:36.742750 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:36.742822 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:36.742832 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:36.742834 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:36.742877 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:45:36.742879 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:36.742918 (+ 39us) write_op.cc:270] Start()
0921 18:45:37.053066 (+310148us) write_op.cc:276] Timestamp: P: 1790016336742898 usec, L: 0
0921 18:45:37.053070 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:37.053299 (+ 229us) log.cc:842] Serialized 354 byte log entry
0921 18:45:38.089253 (+1035954us) op_driver.cc:444] REPLICATION: finished
0921 18:45:38.089497 (+ 244us) write_op.cc:301] APPLY: starting
0921 18:45:38.089515 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:38.089549 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:38.089566 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:38.089627 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:38.089650 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:38.089653 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:38.089702 (+ 49us) log.cc:842] Serialized 22 byte log entry
0921 18:45:38.089788 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:38.089798 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:38.089802 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":167,"num_ops":1,"prepare.queue_time_us":503316,"prepare.run_cpu_time_us":662,"prepare.run_wall_time_us":310731,"replication_time_us":1036160,"thread_start_us":90,"threads_started":1}]]}
I20260921 18:45:38.176904 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:38.177062 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:38.177579 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:38.177951 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:38.177948 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:38.178344 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:38.328047 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:38.329459 6874 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=10, attempt_no=0}) took 1115 ms. Trace:
I20260921 18:45:38.329573 6874 rpcz_store.cc:276] 0921 18:45:37.213589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:37.213649 (+ 60us) service_pool.cc:227] Handling call
0921 18:45:38.329446 (+1115797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:37.213977 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:37.214047 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:37.214049 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:37.214050 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:37.214097 (+ 47us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:37.214102 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:37.214104 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:37.214129 (+ 25us) tablet.cc:699] Row locks acquired
0921 18:45:37.214130 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:37.214155 (+ 25us) write_op.cc:270] Start()
0921 18:45:38.163265 (+949110us) write_op.cc:276] Timestamp: P: 1790016337214093 usec, L: 0
0921 18:45:38.163269 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:38.163484 (+ 215us) log.cc:842] Serialized 354 byte log entry
0921 18:45:38.328905 (+165421us) op_driver.cc:444] REPLICATION: finished
0921 18:45:38.328992 (+ 87us) write_op.cc:301] APPLY: starting
0921 18:45:38.329005 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:38.329084 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:38.329102 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:38.329171 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:38.329192 (+ 21us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:38.329195 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:38.329305 (+ 110us) log.cc:842] Serialized 43 byte log entry
0921 18:45:38.329355 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:38.329365 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:38.329370 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":1,"prepare.queue_time_us":24,"prepare.run_cpu_time_us":544,"prepare.run_wall_time_us":949600,"raft.queue_time_us":44,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":165614,"spinlock_wait_cycles":6016}]]}
I20260921 18:45:38.803838 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:38.803987 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:38.804531 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:38.804916 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:38.804981 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:38.805649 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:39.431623 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:39.431769 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:39.432301 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:39.432713 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:39.432742 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:39.433213 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:39.433403 6884 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=11, attempt_no=0}) took 1342 ms. Trace:
I20260921 18:45:39.433544 6884 rpcz_store.cc:276] 0921 18:45:38.091236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:38.091294 (+ 58us) service_pool.cc:227] Handling call
0921 18:45:39.433389 (+1342095us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:38.163604 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:38.163711 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:38.163714 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:38.163715 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:38.163806 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:38.163816 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:38.163818 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:38.163862 (+ 44us) tablet.cc:699] Row locks acquired
0921 18:45:38.163863 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:38.163906 (+ 43us) write_op.cc:270] Start()
0921 18:45:38.471036 (+307130us) write_op.cc:276] Timestamp: P: 1790016338163848 usec, L: 0
0921 18:45:38.471039 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:38.471190 (+ 151us) log.cc:842] Serialized 354 byte log entry
0921 18:45:39.432787 (+961597us) op_driver.cc:444] REPLICATION: finished
0921 18:45:39.433051 (+ 264us) write_op.cc:301] APPLY: starting
0921 18:45:39.433067 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:39.433095 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:39.433112 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:39.433162 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:39.433190 (+ 28us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:39.433194 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:39.433250 (+ 56us) log.cc:842] Serialized 22 byte log entry
0921 18:45:39.433305 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:39.433316 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:39.433322 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":217,"num_ops":1,"prepare.queue_time_us":71941,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":307656,"raft.queue_time_us":40,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":155,"replication_time_us":961728,"thread_start_us":62,"threads_started":1}]]}
I20260921 18:45:39.898710 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:39.899243 6733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.129:57265 (request call id 2163) took 1094 ms. Trace:
I20260921 18:45:39.899359 6733 rpcz_store.cc:276] 0921 18:45:38.805025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:38.805076 (+ 51us) service_pool.cc:227] Handling call
0921 18:45:38.805210 (+ 134us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:45:38.805279 (+ 69us) raft_consensus.cc:1520] Early marking committed up to index 12
0921 18:45:38.805426 (+ 147us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:45:38.805522 (+ 96us) write_op.cc:270] Start()
0921 18:45:39.775686 (+970164us) write_op.cc:276] Timestamp: P: 1790016338163848 usec, L: 0
0921 18:45:39.775993 (+ 307us) write_op.cc:270] Start()
0921 18:45:39.898164 (+122171us) write_op.cc:276] Timestamp: P: 1790016338471550 usec, L: 0
0921 18:45:39.898459 (+ 295us) log.cc:842] Serialized 708 byte log entry
0921 18:45:39.898617 (+ 158us) raft_consensus.cc:1631] Marking committed up to 12
0921 18:45:39.898623 (+ 6us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:45:39.898671 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:45:39.899167 (+ 496us) raft_consensus.cc:1671] finished
0921 18:45:39.899169 (+ 2us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:45:39.899228 (+ 59us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:39.775992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:39.776084 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:39.776088 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:39.776089 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:39.776150 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:39.776159 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:39.776161 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:39.776196 (+ 35us) tablet.cc:699] Row locks acquired
0921 18:45:39.776197 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0921 18:45:39.898351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:39.898443 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:39.898447 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:39.898448 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:39.898513 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:39.898522 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:39.898524 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:39.898566 (+ 42us) tablet.cc:699] Row locks acquired
0921 18:45:39.898567 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":174,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":270,"prepare.run_wall_time_us":272,"thread_start_us":77,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":298,"prepare.run_wall_time_us":299}]]}
I20260921 18:45:39.901316 6884 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=12, attempt_no=0}) took 1570 ms. Trace:
I20260921 18:45:39.901412 6884 rpcz_store.cc:276] 0921 18:45:38.331089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:38.331144 (+ 55us) service_pool.cc:227] Handling call
0921 18:45:39.901304 (+1570160us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:38.471299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:38.471417 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:38.471421 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:38.471423 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:38.471497 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:38.471505 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:38.471508 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:38.471550 (+ 42us) tablet.cc:699] Row locks acquired
0921 18:45:38.471552 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:38.471591 (+ 39us) write_op.cc:270] Start()
0921 18:45:38.677733 (+206142us) write_op.cc:276] Timestamp: P: 1790016338471550 usec, L: 0
0921 18:45:38.677736 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:38.677918 (+ 182us) log.cc:842] Serialized 354 byte log entry
0921 18:45:39.900829 (+1222911us) op_driver.cc:444] REPLICATION: finished
0921 18:45:39.900940 (+ 111us) write_op.cc:301] APPLY: starting
0921 18:45:39.900954 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:39.900978 (+ 24us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:39.900994 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:39.901070 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:39.901100 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:39.901104 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:39.901167 (+ 63us) log.cc:842] Serialized 22 byte log entry
0921 18:45:39.901231 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:39.901239 (+ 8us) write_op.cc:454] Released schema lock
0921 18:45:39.901244 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"num_ops":1,"prepare.queue_time_us":139798,"prepare.run_cpu_time_us":623,"prepare.run_wall_time_us":206689,"replication_time_us":1223071}]]}
I20260921 18:45:40.056214 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:40.189220 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:40.189361 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:40.189842 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:40.190305 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:40.190332 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:40.190773 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:40.191118 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:41.034384 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:41.034524 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:41.034989 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:41.035431 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:41.035430 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:41.035919 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:41.036083 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:41.169524 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:41.169672 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:41.170208 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:41.170578 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:41.170586 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:41.171087 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:41.171535 6898 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=14, attempt_no=0}) took 1268 ms. Trace:
I20260921 18:45:41.171650 6898 rpcz_store.cc:276] 0921 18:45:39.902809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:39.902898 (+ 89us) service_pool.cc:227] Handling call
0921 18:45:41.171522 (+1268624us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:39.907730 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:39.907825 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:39.907830 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:39.907833 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:39.907938 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:39.907949 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:39.907953 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:39.907987 (+ 34us) tablet.cc:699] Row locks acquired
0921 18:45:39.907988 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:39.908019 (+ 31us) write_op.cc:270] Start()
0921 18:45:40.296176 (+388157us) write_op.cc:276] Timestamp: P: 1790016339907993 usec, L: 0
0921 18:45:40.296180 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:40.296354 (+ 174us) log.cc:842] Serialized 354 byte log entry
0921 18:45:41.170875 (+874521us) op_driver.cc:444] REPLICATION: finished
0921 18:45:41.171157 (+ 282us) write_op.cc:301] APPLY: starting
0921 18:45:41.171172 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:41.171201 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:41.171214 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:41.171264 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:41.171291 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:41.171295 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:41.171359 (+ 64us) log.cc:842] Serialized 43 byte log entry
0921 18:45:41.171407 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:41.171417 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:41.171422 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":234,"num_ops":1,"prepare.queue_time_us":4483,"prepare.run_cpu_time_us":590,"prepare.run_wall_time_us":388679,"replication_time_us":874674,"thread_start_us":69,"threads_started":1}]]}
I20260921 18:45:41.684998 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:41.686812 6903 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=15, attempt_no=0}) took 1494 ms. Trace:
I20260921 18:45:41.686895 6903 rpcz_store.cc:276] 0921 18:45:40.192138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:40.192201 (+ 63us) service_pool.cc:227] Handling call
0921 18:45:41.686796 (+1494595us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:40.296447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:40.296567 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:40.296572 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:40.296574 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:40.296650 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:40.296660 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:40.296662 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:40.296701 (+ 39us) tablet.cc:699] Row locks acquired
0921 18:45:40.296703 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:40.296760 (+ 57us) write_op.cc:270] Start()
0921 18:45:40.970919 (+674159us) write_op.cc:276] Timestamp: P: 1790016340296701 usec, L: 0
0921 18:45:40.970923 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:40.971122 (+ 199us) log.cc:842] Serialized 354 byte log entry
0921 18:45:41.686137 (+715015us) op_driver.cc:444] REPLICATION: finished
0921 18:45:41.686361 (+ 224us) write_op.cc:301] APPLY: starting
0921 18:45:41.686375 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:41.686403 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:41.686418 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:41.686470 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:41.686494 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:41.686497 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:41.686585 (+ 88us) log.cc:842] Serialized 43 byte log entry
0921 18:45:41.686641 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:41.686653 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:41.686705 (+ 52us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":154,"num_ops":1,"prepare.queue_time_us":103901,"prepare.run_cpu_time_us":659,"prepare.run_wall_time_us":674729,"replication_time_us":715192,"thread_start_us":94,"threads_started":1}]]}
I20260921 18:45:41.689352 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:41.689507 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:41.690003 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:41.690338 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:41.690374 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:41.690809 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:41.988420 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:41.988600 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:41.989161 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:41.989570 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:41.989567 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:41.990244 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:41.990403 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:42.078702 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:42.124732 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:42.124866 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:42.125234 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:42.125607 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:42.129866 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:42.130322 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:42.130532 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:42.717783 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:42.717938 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:42.718441 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:42.718818 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:42.718876 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:42.719303 6909 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=17, attempt_no=0}) took 1031 ms. Trace:
I20260921 18:45:42.719468 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:42.719409 6909 rpcz_store.cc:276] 0921 18:45:41.688143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:41.688200 (+ 57us) service_pool.cc:227] Handling call
0921 18:45:42.719288 (+1031088us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:41.688557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:41.688622 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:41.688625 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:41.688626 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:41.688672 (+ 46us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:41.688677 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:41.688679 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:41.688725 (+ 46us) tablet.cc:699] Row locks acquired
0921 18:45:41.688728 (+ 3us) write_op.cc:260] PREPARE: finished
0921 18:45:41.688757 (+ 29us) write_op.cc:270] Start()
0921 18:45:42.216890 (+528133us) write_op.cc:276] Timestamp: P: 1790016341688704 usec, L: 0
0921 18:45:42.216893 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:42.217049 (+ 156us) log.cc:842] Serialized 354 byte log entry
0921 18:45:42.718723 (+501674us) op_driver.cc:444] REPLICATION: finished
0921 18:45:42.718929 (+ 206us) write_op.cc:301] APPLY: starting
0921 18:45:42.718944 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:42.718971 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:42.718984 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:42.719033 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:42.719057 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:42.719061 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:42.719142 (+ 81us) log.cc:842] Serialized 43 byte log entry
0921 18:45:42.719193 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:42.719203 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:42.719208 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":166,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":507,"prepare.run_wall_time_us":528562,"raft.queue_time_us":131,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":287,"replication_time_us":501808,"spinlock_wait_cycles":15616,"thread_start_us":70,"threads_started":1}]]}
I20260921 18:45:42.719681 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:42.802529 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:42.802681 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:42.803275 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:42.803639 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:42.803947 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:42.804422 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:42.804566 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:43.418298 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:43.418412 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:43.419230 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:43.419199 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:43.419298 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:43.419731 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:43.420131 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:43.420459 6917 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=18, attempt_no=0}) took 1429 ms. Trace:
I20260921 18:45:43.420529 6917 rpcz_store.cc:276] 0921 18:45:41.991321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:41.991362 (+ 41us) service_pool.cc:227] Handling call
0921 18:45:43.420433 (+1429071us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:42.217149 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:42.217244 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:42.217248 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:42.217250 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:42.217319 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:42.217327 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:42.217330 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:42.217368 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:45:42.217371 (+ 3us) write_op.cc:260] PREPARE: finished
0921 18:45:42.217404 (+ 33us) write_op.cc:270] Start()
0921 18:45:42.869544 (+652140us) write_op.cc:276] Timestamp: P: 1790016342217349 usec, L: 0
0921 18:45:42.869547 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:42.869722 (+ 175us) log.cc:842] Serialized 354 byte log entry
0921 18:45:43.419790 (+550068us) op_driver.cc:444] REPLICATION: finished
0921 18:45:43.419995 (+ 205us) write_op.cc:301] APPLY: starting
0921 18:45:43.420011 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:43.420038 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:43.420053 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:43.420143 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:43.420170 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:43.420174 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:43.420268 (+ 94us) log.cc:842] Serialized 22 byte log entry
0921 18:45:43.420327 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:43.420340 (+ 13us) write_op.cc:454] Released schema lock
0921 18:45:43.420345 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":162,"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":225442,"prepare.run_cpu_time_us":584,"prepare.run_wall_time_us":652657,"raft.queue_time_us":194,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":550222,"thread_start_us":69,"threads_started":1}]]}
I20260921 18:45:43.818142 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:43.818266 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:43.818768 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:43.819141 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:43.819329 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:43.819734 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:44.122704 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:44.124198 6923 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=19, attempt_no=0}) took 1403 ms. Trace:
I20260921 18:45:44.124277 6923 rpcz_store.cc:276] 0921 18:45:42.720944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:42.720979 (+ 35us) service_pool.cc:227] Handling call
0921 18:45:44.124173 (+1403194us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:42.869836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:42.869948 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:42.869953 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:42.869955 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:42.870031 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:42.870042 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:42.870044 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:42.870085 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:45:42.870086 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:42.870126 (+ 40us) write_op.cc:270] Start()
0921 18:45:43.050274 (+180148us) write_op.cc:276] Timestamp: P: 1790016342870056 usec, L: 0
0921 18:45:43.050277 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:43.050467 (+ 190us) log.cc:842] Serialized 354 byte log entry
0921 18:45:44.123560 (+1073093us) op_driver.cc:444] REPLICATION: finished
0921 18:45:44.123822 (+ 262us) write_op.cc:301] APPLY: starting
0921 18:45:44.123835 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:44.123858 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:44.123871 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:44.123916 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:44.123934 (+ 18us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:44.123937 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:44.124013 (+ 76us) log.cc:842] Serialized 43 byte log entry
0921 18:45:44.124079 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:44.124088 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:44.124092 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":215,"num_ops":1,"prepare.queue_time_us":148485,"prepare.run_cpu_time_us":620,"prepare.run_wall_time_us":180689,"replication_time_us":1073249,"thread_start_us":111,"threads_started":1}]]}
I20260921 18:45:44.253255 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:44.253382 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:44.253861 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:44.254253 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:44.254287 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:44.254673 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:44.254889 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:44.974207 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:44.974354 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:44.974833 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:44.975193 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:44.975271 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:44.975883 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:44.976128 6930 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=20, attempt_no=0}) took 1554 ms. Trace:
I20260921 18:45:44.976207 6930 rpcz_store.cc:276] 0921 18:45:43.421724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:43.421797 (+ 73us) service_pool.cc:227] Handling call
0921 18:45:44.976115 (+1554318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:43.422088 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:43.422154 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:43.422157 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:43.422158 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:43.422220 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:43.422225 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:43.422227 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:43.422251 (+ 24us) tablet.cc:699] Row locks acquired
0921 18:45:43.422252 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:43.422275 (+ 23us) write_op.cc:270] Start()
0921 18:45:44.262389 (+840114us) write_op.cc:276] Timestamp: P: 1790016343422251 usec, L: 0
0921 18:45:44.262392 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:44.262542 (+ 150us) log.cc:842] Serialized 354 byte log entry
0921 18:45:44.975406 (+712864us) op_driver.cc:444] REPLICATION: finished
0921 18:45:44.975584 (+ 178us) write_op.cc:301] APPLY: starting
0921 18:45:44.975601 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:44.975632 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:44.975647 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:44.975700 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:44.975725 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:44.975729 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:44.975805 (+ 76us) log.cc:842] Serialized 43 byte log entry
0921 18:45:44.976015 (+ 210us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:44.976027 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:44.976034 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":127,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":840618,"raft.queue_time_us":167,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":712994,"thread_start_us":158,"threads_started":2}]]}
I20260921 18:45:45.069996 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:45.070151 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:45.070629 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:45.071023 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:45.071080 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:45.071509 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:45.360251 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:45.361588 6930 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=21, attempt_no=0}) took 1236 ms. Trace:
I20260921 18:45:45.361687 6930 rpcz_store.cc:276] 0921 18:45:44.125589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:44.125669 (+ 80us) service_pool.cc:227] Handling call
0921 18:45:45.361574 (+1235905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:44.262731 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:44.262821 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:44.262826 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:44.262828 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:44.262891 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:44.262898 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:44.262900 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:44.262935 (+ 35us) tablet.cc:699] Row locks acquired
0921 18:45:44.262936 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:44.262966 (+ 30us) write_op.cc:270] Start()
0921 18:45:44.268088 (+ 5122us) write_op.cc:276] Timestamp: P: 1790016344262909 usec, L: 0
0921 18:45:44.268106 (+ 18us) op_driver.cc:331] REPLICATION: starting
0921 18:45:44.268234 (+ 128us) log.cc:842] Serialized 354 byte log entry
0921 18:45:45.361027 (+1092793us) op_driver.cc:444] REPLICATION: finished
0921 18:45:45.361135 (+ 108us) write_op.cc:301] APPLY: starting
0921 18:45:45.361154 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:45.361184 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:45.361202 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:45.361263 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:45.361293 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:45.361296 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:45.361393 (+ 97us) log.cc:842] Serialized 43 byte log entry
0921 18:45:45.361458 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:45.361488 (+ 30us) write_op.cc:454] Released schema lock
0921 18:45:45.361493 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":1,"prepare.queue_time_us":136688,"prepare.run_cpu_time_us":480,"prepare.run_wall_time_us":5550,"replication_time_us":1092892}]]}
I20260921 18:45:45.399700 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:45.474421 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:45.474622 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:45.475159 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:45.475590 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:45.475647 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:45.475980 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:45.476222 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:45.834409 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:45.834540 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:45.835062 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:45.835461 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:45.835575 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:45.836107 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:45.836858 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:46.004696 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:46.004840 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:46.005368 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:46.005769 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:46.005854 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:46.006258 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:46.542081 6827 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:46.542250 6827 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:46.542879 6827 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:46.543246 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:46.543335 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:46.543800 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:46.544021 6827 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:46.544348 6940 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=24, attempt_no=0}) took 1067 ms. Trace:
I20260921 18:45:46.544445 6940 rpcz_store.cc:276] 0921 18:45:45.477036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:45.477090 (+ 54us) service_pool.cc:227] Handling call
0921 18:45:46.544334 (+1067244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:45.719061 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:45.719203 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:45.719207 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:45.719208 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:45.719277 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:45.719285 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:45.719287 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:45.719317 (+ 30us) tablet.cc:699] Row locks acquired
0921 18:45:45.719318 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:45.719351 (+ 33us) write_op.cc:270] Start()
0921 18:45:45.850498 (+131147us) write_op.cc:276] Timestamp: P: 1790016345719304 usec, L: 0
0921 18:45:45.850500 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:45:45.850611 (+ 111us) log.cc:842] Serialized 354 byte log entry
0921 18:45:46.543554 (+692943us) op_driver.cc:444] REPLICATION: finished
0921 18:45:46.543773 (+ 219us) write_op.cc:301] APPLY: starting
0921 18:45:46.543788 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:46.543814 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:46.543828 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:46.543909 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:46.543938 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:46.543942 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:46.544011 (+ 69us) log.cc:842] Serialized 43 byte log entry
0921 18:45:46.544234 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:46.544253 (+ 19us) write_op.cc:454] Released schema lock
0921 18:45:46.544258 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":171,"num_ops":1,"prepare.queue_time_us":241680,"prepare.run_cpu_time_us":556,"prepare.run_wall_time_us":131639,"raft.queue_time_us":40,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":300,"replication_time_us":693034,"spinlock_wait_cycles":51968,"thread_start_us":67,"threads_started":1}]]}
I20260921 18:45:47.587930 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:47.588234 6733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.3.36.129:57265 (request call id 2521) took 1582 ms. Trace:
I20260921 18:45:47.588312 6733 rpcz_store.cc:276] 0921 18:45:46.005848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:46.005945 (+ 97us) service_pool.cc:227] Handling call
0921 18:45:46.006086 (+ 141us) raft_consensus.cc:1457] Updating replica for 2 ops
0921 18:45:46.006165 (+ 79us) raft_consensus.cc:1520] Early marking committed up to index 24
0921 18:45:46.006300 (+ 135us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0921 18:45:46.006369 (+ 69us) write_op.cc:270] Start()
0921 18:45:46.826541 (+820172us) write_op.cc:276] Timestamp: P: 1790016345363466 usec, L: 0
0921 18:45:46.826873 (+ 332us) write_op.cc:270] Start()
0921 18:45:47.587037 (+760164us) write_op.cc:276] Timestamp: P: 1790016345719304 usec, L: 0
0921 18:45:47.587549 (+ 512us) log.cc:842] Serialized 708 byte log entry
0921 18:45:47.587699 (+ 150us) raft_consensus.cc:1631] Marking committed up to 25
0921 18:45:47.587834 (+ 135us) raft_consensus.cc:1683] Filling consensus response to leader.
0921 18:45:47.587864 (+ 30us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0921 18:45:47.588161 (+ 297us) raft_consensus.cc:1671] finished
0921 18:45:47.588164 (+ 3us) raft_consensus.cc:1677] UpdateReplicas() finished
0921 18:45:47.588216 (+ 52us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:46.826855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:46.826953 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:46.826960 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:45:46.826963 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:46.827053 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:46.827065 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:46.827068 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:46.827111 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:45:46.827112 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:47.587729 (+760617us) op_driver.cc:444] REPLICATION: finished
0921 18:45:47.587936 (+ 207us) write_op.cc:301] APPLY: starting
0921 18:45:47.587952 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:47.587982 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:47.587983 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:47.588211 (+ 228us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:47.588237 (+ 26us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:47.588253 (+ 16us) write_op.cc:312] APPLY: finished
0921 18:45:47.588301 (+ 48us) log.cc:842] Serialized 22 byte log entry
0921 18:45:47.588372 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:47.588381 (+ 9us) write_op.cc:454] Released schema lock
Related trace 'op':
0921 18:45:47.587426 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:47.587560 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:47.587563 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:47.587565 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:47.587628 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:47.587637 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:47.587639 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:47.587672 (+ 33us) tablet.cc:699] Row locks acquired
0921 18:45:47.587673 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":134,"child_traces":[["op",{"apply.queue_time_us":152,"apply.run_cpu_time_us":560,"apply.run_wall_time_us":561,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":200,"prepare.run_cpu_time_us":327,"prepare.run_wall_time_us":326,"replication_time_us":1581347,"spinlock_wait_cycles":6016,"thread_start_us":156,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":248,"prepare.run_cpu_time_us":360,"prepare.run_wall_time_us":364,"thread_start_us":82,"threads_started":1}]]}
I20260921 18:45:47.810617 6943 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:47.810792 6943 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:47.811317 6943 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:47.811697 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:47.811833 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:47.812288 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:47.812433 6943 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:47.812696 6951 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=25, attempt_no=0}) took 1975 ms. Trace:
I20260921 18:45:47.812773 6951 rpcz_store.cc:276] 0921 18:45:45.837000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:45.837055 (+ 55us) service_pool.cc:227] Handling call
0921 18:45:47.812681 (+1975626us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:45.850717 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:45.850800 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:45.850803 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:45.850805 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:45.850858 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:45.850864 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:45.850867 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:46.544318 (+693451us) lock_manager.cc:599] Waited 693303us for lock on <redacted> (key hash 18178737835805687895)
0921 18:45:46.544324 (+ 6us) tablet.cc:699] Row locks acquired
0921 18:45:46.544326 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:46.544373 (+ 47us) write_op.cc:270] Start()
0921 18:45:47.060497 (+516124us) write_op.cc:276] Timestamp: P: 1790016346544343 usec, L: 0
0921 18:45:47.060500 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:47.060643 (+ 143us) log.cc:842] Serialized 354 byte log entry
0921 18:45:47.811793 (+751150us) op_driver.cc:444] REPLICATION: finished
0921 18:45:47.812289 (+ 496us) write_op.cc:301] APPLY: starting
0921 18:45:47.812319 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:47.812346 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:47.812348 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:47.812404 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:47.812434 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:47.812438 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:47.812525 (+ 87us) log.cc:842] Serialized 43 byte log entry
0921 18:45:47.812580 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:47.812602 (+ 22us) write_op.cc:454] Released schema lock
0921 18:45:47.812607 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":441,"num_ops":1,"prepare.queue_time_us":13356,"prepare.run_cpu_time_us":677,"prepare.run_wall_time_us":1209996,"replication_time_us":751274,"row_lock_wait_count":1,"row_lock_wait_us":693303,"thread_start_us":56,"threads_started":1}]]}
I20260921 18:45:47.828553 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:47.828665 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:47.829282 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:47.829716 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:47.829716 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:47.830221 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:47.830365 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:48.344666 6943 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:48.344794 6943 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:48.345288 6943 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:48.345633 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:48.345798 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:48.346140 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:48.346169 6954 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=26, attempt_no=0}) took 1800 ms. Trace:
I20260921 18:45:48.346333 6943 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:48.346271 6954 rpcz_store.cc:276] 0921 18:45:46.545755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:46.545822 (+ 67us) service_pool.cc:227] Handling call
0921 18:45:48.346153 (+1800331us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:47.060746 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:47.060850 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:47.060854 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:47.060856 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:47.060928 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:47.060940 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:47.060943 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:47.812610 (+751667us) lock_manager.cc:599] Waited 751610us for lock on <redacted> (key hash 18178737835805687895)
0921 18:45:47.812616 (+ 6us) tablet.cc:699] Row locks acquired
0921 18:45:47.812617 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:47.812684 (+ 67us) write_op.cc:270] Start()
0921 18:45:47.921831 (+109147us) write_op.cc:276] Timestamp: P: 1790016347812629 usec, L: 0
0921 18:45:47.921834 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:47.922013 (+ 179us) log.cc:842] Serialized 354 byte log entry
0921 18:45:48.345569 (+423556us) op_driver.cc:444] REPLICATION: finished
0921 18:45:48.345774 (+ 205us) write_op.cc:301] APPLY: starting
0921 18:45:48.345802 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:48.345831 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:48.345833 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:48.345892 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:48.345917 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:48.345920 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:48.345995 (+ 75us) log.cc:842] Serialized 43 byte log entry
0921 18:45:48.346047 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:48.346058 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:48.346063 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":156,"num_ops":1,"prepare.queue_time_us":514636,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":861329,"replication_time_us":423712,"row_lock_wait_count":1,"row_lock_wait_us":751610,"spinlock_wait_cycles":77184,"thread_start_us":89,"threads_started":1}]]}
I20260921 18:45:48.704862 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:48.704998 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:48.705654 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:48.706012 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:48.706224 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:48.706591 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:48.968164 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:48.969894 6959 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45024 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=27, attempt_no=0}) took 1155 ms. Trace:
I20260921 18:45:48.969996 6959 rpcz_store.cc:276] 0921 18:45:47.814280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:47.814356 (+ 76us) service_pool.cc:227] Handling call
0921 18:45:48.969877 (+1155521us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:47.922110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:47.922216 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:47.922220 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:47.922222 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:47.922296 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:47.922304 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:47.922307 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:47.922343 (+ 36us) tablet.cc:699] Row locks acquired
0921 18:45:47.922345 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:47.922385 (+ 40us) write_op.cc:270] Start()
0921 18:45:48.621554 (+699169us) write_op.cc:276] Timestamp: P: 1790016347922317 usec, L: 0
0921 18:45:48.621558 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:48.621709 (+ 151us) log.cc:842] Serialized 354 byte log entry
0921 18:45:48.969217 (+347508us) op_driver.cc:444] REPLICATION: finished
0921 18:45:48.969463 (+ 246us) write_op.cc:301] APPLY: starting
0921 18:45:48.969478 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:48.969537 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:48.969553 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:48.969612 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:48.969659 (+ 47us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:48.969662 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:48.969720 (+ 58us) log.cc:842] Serialized 22 byte log entry
0921 18:45:48.969794 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:48.969805 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:48.969809 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":173,"num_ops":1,"prepare.queue_time_us":107412,"prepare.run_cpu_time_us":562,"prepare.run_wall_time_us":699663,"raft.queue_time_us":18,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"replication_time_us":347639,"spinlock_wait_cycles":16256,"thread_start_us":80,"threads_started":1}]]}
I20260921 18:45:49.009616 6862 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:49.009750 6862 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.010293 6862 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:49.010648 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:49.010671 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:49.011127 6667 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c, 7a46f3a6b1634626be69e0782c521f15
I20260921 18:45:49.011409 6862 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:49.078971 6943 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:49.079113 6943 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.079622 6943 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:49.079910 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:49.079957 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:49.080343 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:49.080492 6943 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:49.187532 6943 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:49.187664 6943 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.188134 6943 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:49.188505 6733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:49.188587 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:49.188946 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:49.189126 6943 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:45:49.556851 6962 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:49.557019 6962 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.557389 6962 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:49.557760 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:49.557886 6601 raft_consensus.cc:2454] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3f9f8b70dc432699d59db2668757d9 in term 1.
I20260921 18:45:49.558187 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9; no voters:
I20260921 18:45:49.558346 6962 raft_consensus.cc:2781] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260921 18:45:49.558439 6962 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3ed466b1d5d14005adcb29914704ab7c)
I20260921 18:45:49.558501 6962 raft_consensus.cc:3037] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:45:49.559371 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:49.559718 6962 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.560040 6962 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 election: Requested vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:49.560249 6732 raft_consensus.cc:1226] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 2 FOLLOWER]: Rejecting Update request from peer 3ed466b1d5d14005adcb29914704ab7c for earlier term 1. Current term is 2. Ops: []
I20260921 18:45:49.560436 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15"
I20260921 18:45:49.560467 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "dd3f9f8b70dc432699d59db2668757d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c"
I20260921 18:45:49.560549 6601 raft_consensus.cc:3037] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:45:49.561264 6947 consensus_queue.cc:1046] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 }, Status: INVALID_TERM, Last received: 1.30, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260921 18:45:49.561640 6958 raft_consensus.cc:3032] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:45:49.561692 6958 raft_consensus.cc:729] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3ed466b1d5d14005adcb29914704ab7c, State: Running, Role: LEADER
I20260921 18:45:49.561689 6601 raft_consensus.cc:2454] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3f9f8b70dc432699d59db2668757d9 in term 2.
I20260921 18:45:49.561865 6602 raft_consensus.cc:1226] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Rejecting Update request from peer 3ed466b1d5d14005adcb29914704ab7c for earlier term 1. Current term is 2. Ops: []
I20260921 18:45:49.561861 6958 consensus_queue.cc:260] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.562004 6958 raft_consensus.cc:3037] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:45:49.562608 6666 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9; no voters: 3ed466b1d5d14005adcb29914704ab7c
I20260921 18:45:49.562749 6962 raft_consensus.cc:2781] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 2 FOLLOWER]: Leader election won for term 2
I20260921 18:45:49.562906 6962 raft_consensus.cc:686] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [term 2 LEADER]: Becoming Leader. State: Replica: dd3f9f8b70dc432699d59db2668757d9, State: Running, Role: LEADER
I20260921 18:45:49.563231 6962 consensus_queue.cc:237] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:49.565541 6602 raft_consensus.cc:1261] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Refusing update from remote peer dd3f9f8b70dc432699d59db2668757d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20260921 18:45:49.565515 6329 catalog_manager.cc:5495] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 reported cstate change: term changed from 1 to 2, leader changed from 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129) to dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131). New cstate: current_term: 2 leader_uuid: "dd3f9f8b70dc432699d59db2668757d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260921 18:45:49.565873 6471 raft_consensus.cc:1261] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Refusing update from remote peer dd3f9f8b70dc432699d59db2668757d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20260921 18:45:49.566042 6963 consensus_queue.cc:1035] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20260921 18:45:49.566380 6962 consensus_queue.cc:1035] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20260921 18:45:49.887624 6932 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45024: Illegal state: Replica 3ed466b1d5d14005adcb29914704ab7c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "dd3f9f8b70dc432699d59db2668757d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } } }
W20260921 18:45:49.888525 6932 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45024: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.000s ago.
I20260921 18:45:51.004199 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:51.004359 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:51.004886 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:51.005332 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:51.005363 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:51.005820 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:51.006027 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:51.006537 6979 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=30, attempt_no=1}) took 1110 ms. Trace:
I20260921 18:45:51.006597 6979 rpcz_store.cc:276] 0921 18:45:49.895656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:49.895703 (+ 47us) service_pool.cc:227] Handling call
0921 18:45:51.006527 (+1110824us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:49.896695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:49.896780 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:49.896784 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:49.896785 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:49.896845 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:49.896862 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:49.896866 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:49.896904 (+ 38us) tablet.cc:699] Row locks acquired
0921 18:45:49.896905 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:49.896950 (+ 45us) write_op.cc:270] Start()
0921 18:45:50.331089 (+434139us) write_op.cc:276] Timestamp: P: 1790016349896924 usec, L: 0
0921 18:45:50.331094 (+ 5us) op_driver.cc:331] REPLICATION: starting
0921 18:45:50.331280 (+ 186us) log.cc:842] Serialized 354 byte log entry
0921 18:45:51.005609 (+674329us) op_driver.cc:444] REPLICATION: finished
0921 18:45:51.005998 (+ 389us) write_op.cc:301] APPLY: starting
0921 18:45:51.006014 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:51.006043 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:51.006060 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:51.006155 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:51.006179 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:51.006188 (+ 9us) write_op.cc:312] APPLY: finished
0921 18:45:51.006252 (+ 64us) log.cc:842] Serialized 43 byte log entry
0921 18:45:51.006297 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:51.006306 (+ 9us) write_op.cc:454] Released schema lock
0921 18:45:51.006310 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":53120,"child_traces":[["op",{"apply.queue_time_us":132,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":603,"prepare.run_wall_time_us":434667,"raft.queue_time_us":36,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":192,"replication_time_us":674497,"thread_start_us":72,"threads_started":1}]]}
I20260921 18:45:51.266884 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:51.267021 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:51.267824 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:51.268131 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:51.268221 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:51.268673 6405 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:51.281883 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:51.282011 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:51.282490 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:51.282842 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:51.282874 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:51.283373 6979 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=29, attempt_no=1}) took 1387 ms. Trace:
I20260921 18:45:51.283453 6979 rpcz_store.cc:276] 0921 18:45:49.895777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:49.895836 (+ 59us) service_pool.cc:227] Handling call
0921 18:45:51.283351 (+1387515us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:50.331396 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:50.331502 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:50.331507 (+ 5us) write_op.cc:435] Acquired schema lock
0921 18:45:50.331508 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:50.331582 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:50.331594 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:50.331596 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:50.331649 (+ 53us) tablet.cc:699] Row locks acquired
0921 18:45:50.331650 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:50.331688 (+ 38us) write_op.cc:270] Start()
0921 18:45:50.748805 (+417117us) write_op.cc:276] Timestamp: P: 1790016350331660 usec, L: 0
0921 18:45:50.748809 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:50.749003 (+ 194us) log.cc:842] Serialized 354 byte log entry
0921 18:45:51.282639 (+533636us) op_driver.cc:444] REPLICATION: finished
0921 18:45:51.282992 (+ 353us) write_op.cc:301] APPLY: starting
0921 18:45:51.283006 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:51.283033 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:51.283047 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:51.283098 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:51.283122 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:51.283126 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:51.283210 (+ 84us) log.cc:842] Serialized 43 byte log entry
0921 18:45:51.283263 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:51.283276 (+ 13us) write_op.cc:454] Released schema lock
0921 18:45:51.283281 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":553088,"child_traces":[["op",{"apply.queue_time_us":32,"mutex_wait_us":54,"num_ops":1,"prepare.queue_time_us":434750,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":417694,"replication_time_us":533797}]]}
I20260921 18:45:51.283653 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:51.283938 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:51.431092 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:51.653900 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:51.654039 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:51.654541 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:51.654918 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:51.655155 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:51.655548 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:51.655712 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:52.339524 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:52.339689 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:52.340190 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:52.340611 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 35 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:52.340693 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 35 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:52.341107 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:52.341190 6987 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=32, attempt_no=0}) took 1056 ms. Trace:
I20260921 18:45:52.341259 6987 rpcz_store.cc:276] 0921 18:45:51.284918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:51.284974 (+ 56us) service_pool.cc:227] Handling call
0921 18:45:52.341177 (+1056203us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:51.328728 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:51.328822 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:51.328828 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:45:51.328831 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:51.328899 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:51.328909 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:51.328914 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:51.328959 (+ 45us) tablet.cc:699] Row locks acquired
0921 18:45:51.328960 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:51.328990 (+ 30us) write_op.cc:270] Start()
0921 18:45:51.672137 (+343147us) write_op.cc:276] Timestamp: P: 1790016351328924 usec, L: 0
0921 18:45:51.672140 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:51.672292 (+ 152us) log.cc:842] Serialized 354 byte log entry
0921 18:45:52.340616 (+668324us) op_driver.cc:444] REPLICATION: finished
0921 18:45:52.340830 (+ 214us) write_op.cc:301] APPLY: starting
0921 18:45:52.340846 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:52.340877 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:52.340895 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:52.340952 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:52.340983 (+ 31us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:52.340987 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:52.341048 (+ 61us) log.cc:842] Serialized 22 byte log entry
0921 18:45:52.341099 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:52.341110 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:52.341114 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":173,"num_ops":1,"prepare.queue_time_us":43434,"prepare.run_cpu_time_us":557,"prepare.run_wall_time_us":343629,"replication_time_us":668455,"thread_start_us":67,"threads_started":1}]]}
I20260921 18:45:52.366189 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:52.366312 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:52.366798 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:52.367136 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:52.367231 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:52.367622 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:52.604833 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:52.990212 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:53.313252 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:53.313398 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:53.313972 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:53.314381 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:53.314401 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:53.314929 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:53.315166 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:53.430069 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:53.430202 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:53.430706 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:53.431145 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:53.431180 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:53.431581 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:53.431833 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:53.738946 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:53.739095 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:53.739566 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:53.739910 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:53.739953 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:53.740334 6405 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:53.740512 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:54.144590 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:54.144734 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:54.145263 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:54.145758 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:54.146088 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:54.146479 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:54.498996 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:54.499137 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:54.499655 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:54.500139 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:54.500211 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:54.500643 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:54.516270 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:54.517740 7002 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=37, attempt_no=0}) took 1084 ms. Trace:
I20260921 18:45:54.517823 7002 rpcz_store.cc:276] 0921 18:45:53.433019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:53.433064 (+ 45us) service_pool.cc:227] Handling call
0921 18:45:54.517724 (+1084660us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:53.649664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:53.649763 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:53.649767 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:53.649769 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:53.649840 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:53.649849 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:53.649852 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:53.649893 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:45:53.649895 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:53.649934 (+ 39us) write_op.cc:270] Start()
0921 18:45:53.993078 (+343144us) write_op.cc:276] Timestamp: P: 1790016353649882 usec, L: 0
0921 18:45:53.993082 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:53.993264 (+ 182us) log.cc:842] Serialized 354 byte log entry
0921 18:45:54.517219 (+523955us) op_driver.cc:444] REPLICATION: finished
0921 18:45:54.517301 (+ 82us) write_op.cc:301] APPLY: starting
0921 18:45:54.517314 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:54.517359 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:54.517376 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:54.517429 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:54.517463 (+ 34us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:54.517488 (+ 25us) write_op.cc:312] APPLY: finished
0921 18:45:54.517555 (+ 67us) log.cc:842] Serialized 43 byte log entry
0921 18:45:54.517632 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:54.517642 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:54.517647 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":25,"num_ops":1,"prepare.queue_time_us":216313,"prepare.run_cpu_time_us":593,"prepare.run_wall_time_us":343653,"replication_time_us":524114}]]}
I20260921 18:45:55.026907 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:55.312894 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:55.313087 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:55.313601 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:55.314016 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:55.314028 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:55.314455 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:55.314657 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:55.314929 7011 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=38, attempt_no=0}) took 1167 ms. Trace:
I20260921 18:45:55.315011 7011 rpcz_store.cc:276] 0921 18:45:54.147275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:54.147330 (+ 55us) service_pool.cc:227] Handling call
0921 18:45:55.314915 (+1167585us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:54.147655 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:54.147720 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:54.147723 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:45:54.147724 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:54.147771 (+ 47us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:54.147776 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:54.147777 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:54.147801 (+ 24us) tablet.cc:699] Row locks acquired
0921 18:45:54.147802 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:54.147826 (+ 24us) write_op.cc:270] Start()
0921 18:45:54.511946 (+364120us) write_op.cc:276] Timestamp: P: 1790016354147790 usec, L: 0
0921 18:45:54.511950 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:54.512085 (+ 135us) log.cc:842] Serialized 354 byte log entry
0921 18:45:55.313908 (+801823us) op_driver.cc:444] REPLICATION: finished
0921 18:45:55.314109 (+ 201us) write_op.cc:301] APPLY: starting
0921 18:45:55.314123 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:55.314588 (+ 465us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:55.314602 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:55.314657 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:55.314686 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:55.314691 (+ 5us) write_op.cc:312] APPLY: finished
0921 18:45:55.314765 (+ 74us) log.cc:842] Serialized 43 byte log entry
0921 18:45:55.314818 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:55.314828 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:55.314833 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":159,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":421,"prepare.run_wall_time_us":364485,"replication_time_us":801935,"thread_start_us":59,"threads_started":1}]]}
I20260921 18:45:55.543200 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:55.543342 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:55.543844 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:55.544162 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:55.544246 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:55.544684 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:56.349732 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:56.351279 7018 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=39, attempt_no=0}) took 1831 ms. Trace:
I20260921 18:45:56.351397 7018 rpcz_store.cc:276] 0921 18:45:54.519842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:54.519909 (+ 67us) service_pool.cc:227] Handling call
0921 18:45:56.351264 (+1831355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:54.520326 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:54.520428 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:54.520430 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:54.520441 (+ 11us) tablet.cc:660] Decoding operations
0921 18:45:54.520499 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:54.520505 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:54.520508 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:54.520557 (+ 49us) tablet.cc:699] Row locks acquired
0921 18:45:54.520559 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:45:54.520591 (+ 32us) write_op.cc:270] Start()
0921 18:45:55.444711 (+924120us) write_op.cc:276] Timestamp: P: 1790016354520556 usec, L: 0
0921 18:45:55.444714 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:55.444899 (+ 185us) log.cc:842] Serialized 354 byte log entry
0921 18:45:56.350693 (+905794us) op_driver.cc:444] REPLICATION: finished
0921 18:45:56.350875 (+ 182us) write_op.cc:301] APPLY: starting
0921 18:45:56.350888 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:56.350911 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:56.350926 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:56.350973 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:56.350992 (+ 19us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:56.350994 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:56.351116 (+ 122us) log.cc:842] Serialized 43 byte log entry
0921 18:45:56.351171 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:56.351181 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:56.351186 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":128,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":924651,"replication_time_us":905960,"thread_start_us":79,"threads_started":1}]]}
I20260921 18:45:56.388077 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:56.388221 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:56.388684 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:56.389046 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:56.389046 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:56.389513 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:56.960245 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:56.961814 7026 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=40, attempt_no=0}) took 1645 ms. Trace:
I20260921 18:45:56.961908 7026 rpcz_store.cc:276] 0921 18:45:55.316161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:55.316201 (+ 40us) service_pool.cc:227] Handling call
0921 18:45:56.961775 (+1645574us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:55.445003 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:55.445106 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:55.445110 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:55.445112 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:55.445185 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:55.445196 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:55.445198 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:55.445241 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:45:55.445242 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:55.445273 (+ 31us) write_op.cc:270] Start()
0921 18:45:55.882400 (+437127us) write_op.cc:276] Timestamp: P: 1790016355445251 usec, L: 0
0921 18:45:55.882403 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:55.882586 (+ 183us) log.cc:842] Serialized 354 byte log entry
0921 18:45:56.961239 (+1078653us) op_driver.cc:444] REPLICATION: finished
0921 18:45:56.961400 (+ 161us) write_op.cc:301] APPLY: starting
0921 18:45:56.961412 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:56.961435 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:56.961449 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:56.961518 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:56.961536 (+ 18us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:56.961538 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:45:56.961628 (+ 90us) log.cc:842] Serialized 43 byte log entry
0921 18:45:56.961680 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:56.961691 (+ 11us) write_op.cc:454] Released schema lock
0921 18:45:56.961695 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"num_ops":1,"prepare.queue_time_us":128522,"prepare.run_cpu_time_us":598,"prepare.run_wall_time_us":437658,"replication_time_us":1078817,"thread_start_us":61,"threads_started":1}]]}
I20260921 18:45:57.293260 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:57.293416 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:57.293943 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:57.294312 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:57.294378 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:57.294749 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:57.662595 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:57.664247 7031 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=41, attempt_no=0}) took 1311 ms. Trace:
I20260921 18:45:57.664395 7031 rpcz_store.cc:276] 0921 18:45:56.352656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:56.352716 (+ 60us) service_pool.cc:227] Handling call
0921 18:45:57.664235 (+1311519us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:56.353055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:56.353118 (+ 63us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:56.353120 (+ 2us) write_op.cc:435] Acquired schema lock
0921 18:45:56.353122 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:56.353169 (+ 47us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:56.353174 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:56.353175 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:56.353201 (+ 26us) tablet.cc:699] Row locks acquired
0921 18:45:56.353201 (+ 0us) write_op.cc:260] PREPARE: finished
0921 18:45:56.353225 (+ 24us) write_op.cc:270] Start()
0921 18:45:57.151349 (+798124us) write_op.cc:276] Timestamp: P: 1790016356353175 usec, L: 0
0921 18:45:57.151353 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:45:57.151489 (+ 136us) log.cc:842] Serialized 354 byte log entry
0921 18:45:57.663569 (+512080us) op_driver.cc:444] REPLICATION: finished
0921 18:45:57.663807 (+ 238us) write_op.cc:301] APPLY: starting
0921 18:45:57.663822 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:57.663849 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:57.663861 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:57.663919 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:57.663949 (+ 30us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:57.663953 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:57.664064 (+ 111us) log.cc:842] Serialized 43 byte log entry
0921 18:45:57.664135 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:57.664147 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:57.664153 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":195,"num_ops":1,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":473,"prepare.run_wall_time_us":798539,"raft.queue_time_us":58,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"replication_time_us":512195,"thread_start_us":74,"threads_started":1}]]}
I20260921 18:45:57.680797 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:57.680934 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:57.681412 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:57.681777 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:57.682061 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:57.682518 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:57.682858 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:58.085706 6973 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:58.085876 6973 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:58.086403 6973 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:58.086700 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:58.086777 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:58.087075 7031 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=42, attempt_no=0}) took 1123 ms. Trace:
I20260921 18:45:58.087225 6405 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:58.087154 7031 rpcz_store.cc:276] 0921 18:45:56.963210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:56.963265 (+ 55us) service_pool.cc:227] Handling call
0921 18:45:58.087062 (+1123797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:57.151605 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:57.151705 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:57.151709 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:57.151711 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:57.151781 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:57.151789 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:57.151792 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:57.151832 (+ 40us) tablet.cc:699] Row locks acquired
0921 18:45:57.151833 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:57.151873 (+ 40us) write_op.cc:270] Start()
0921 18:45:58.009003 (+857130us) write_op.cc:276] Timestamp: P: 1790016357151840 usec, L: 0
0921 18:45:58.009006 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:58.009212 (+ 206us) log.cc:842] Serialized 354 byte log entry
0921 18:45:58.086599 (+ 77387us) op_driver.cc:444] REPLICATION: finished
0921 18:45:58.086686 (+ 87us) write_op.cc:301] APPLY: starting
0921 18:45:58.086701 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:58.086728 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:58.086744 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:58.086802 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:58.086829 (+ 27us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:58.086833 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:58.086899 (+ 66us) log.cc:842] Serialized 43 byte log entry
0921 18:45:58.086948 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:58.086958 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:58.086963 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":187988,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":857683,"raft.queue_time_us":188,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":179,"replication_time_us":77573}]]}
I20260921 18:45:58.087388 6973 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:58.666225 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:58.666389 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:58.666909 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:58.667248 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:58.667341 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:58.667811 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:58.667964 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:59.129272 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:59.129456 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:59.129931 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:45:59.130358 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 46 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:45:59.130486 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 46 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:59.130975 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:59.131354 7038 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=43, attempt_no=0}) took 1465 ms. Trace:
I20260921 18:45:59.131436 7038 rpcz_store.cc:276] 0921 18:45:57.665795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:57.665834 (+ 39us) service_pool.cc:227] Handling call
0921 18:45:59.131339 (+1465505us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:58.009326 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:58.009427 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:58.009433 (+ 6us) write_op.cc:435] Acquired schema lock
0921 18:45:58.009436 (+ 3us) tablet.cc:660] Decoding operations
0921 18:45:58.009531 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:58.009539 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:58.009542 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:58.009583 (+ 41us) tablet.cc:699] Row locks acquired
0921 18:45:58.009584 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:58.009613 (+ 29us) write_op.cc:270] Start()
0921 18:45:58.669765 (+660152us) write_op.cc:276] Timestamp: P: 1790016358009590 usec, L: 0
0921 18:45:58.669768 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:58.669971 (+ 203us) log.cc:842] Serialized 354 byte log entry
0921 18:45:59.130237 (+460266us) op_driver.cc:444] REPLICATION: finished
0921 18:45:59.130921 (+ 684us) write_op.cc:301] APPLY: starting
0921 18:45:59.130969 (+ 48us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:59.131000 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:59.131014 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:59.131070 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:59.131095 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:59.131099 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:45:59.131191 (+ 92us) log.cc:842] Serialized 43 byte log entry
0921 18:45:59.131243 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:59.131255 (+ 12us) write_op.cc:454] Released schema lock
0921 18:45:59.131261 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":629,"num_ops":1,"prepare.queue_time_us":343136,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":660726,"raft.queue_time_us":37,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"replication_time_us":460446,"spinlock_wait_cycles":137856,"thread_start_us":78,"threads_started":1}]]}
I20260921 18:45:59.357280 7037 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:45:59.357414 7037 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:45:59.358011 7037 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:45:59.358346 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 46 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:45:59.358430 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 46 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:45:59.359156 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:45:59.380685 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:45:59.381934 7038 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=44, attempt_no=0}) took 1293 ms. Trace:
I20260921 18:45:59.382028 7038 rpcz_store.cc:276] 0921 18:45:58.088642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:58.088685 (+ 43us) service_pool.cc:227] Handling call
0921 18:45:59.381921 (+1293236us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:58.670085 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:58.670164 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:58.670168 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:58.670169 (+ 1us) tablet.cc:660] Decoding operations
0921 18:45:58.670232 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:58.670240 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:58.670243 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:58.670277 (+ 34us) tablet.cc:699] Row locks acquired
0921 18:45:58.670278 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:58.670309 (+ 31us) write_op.cc:270] Start()
0921 18:45:58.785438 (+115129us) write_op.cc:276] Timestamp: P: 1790016358670265 usec, L: 0
0921 18:45:58.785441 (+ 3us) op_driver.cc:331] REPLICATION: starting
0921 18:45:58.785699 (+ 258us) log.cc:842] Serialized 354 byte log entry
0921 18:45:59.381440 (+595741us) op_driver.cc:444] REPLICATION: finished
0921 18:45:59.381558 (+ 118us) write_op.cc:301] APPLY: starting
0921 18:45:59.381573 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:45:59.381603 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0921 18:45:59.381615 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:45:59.381664 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:45:59.381682 (+ 18us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:45:59.381685 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:45:59.381765 (+ 80us) log.cc:842] Serialized 43 byte log entry
0921 18:45:59.381830 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:45:59.381840 (+ 10us) write_op.cc:454] Released schema lock
0921 18:45:59.381846 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"num_ops":1,"prepare.queue_time_us":581054,"prepare.run_cpu_time_us":594,"prepare.run_wall_time_us":115666,"replication_time_us":595982}]]}
I20260921 18:46:00.101723 7037 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:00.131033 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:00.131178 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:00.131596 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:46:00.132112 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:46:00.132169 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:00.132647 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:00.132874 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:00.262914 7037 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:00.263075 7037 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:00.263590 7037 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:46:00.263862 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:00.263859 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:46:00.264562 6405 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:00.264725 7037 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:00.422214 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:00.422345 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:00.422775 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:46:00.423069 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:46:00.423146 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:00.423571 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:00.423830 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:01.043793 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:01.043926 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:01.044365 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:46:01.044823 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 49 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:46:01.045094 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 49 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:01.045501 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:01.045732 7049 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=46, attempt_no=0}) took 1662 ms. Trace:
I20260921 18:46:01.045805 7049 rpcz_store.cc:276] 0921 18:45:59.383319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:45:59.383375 (+ 56us) service_pool.cc:227] Handling call
0921 18:46:01.045669 (+1662294us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:45:59.637788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:45:59.637909 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:45:59.637913 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:45:59.637915 (+ 2us) tablet.cc:660] Decoding operations
0921 18:45:59.638001 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:45:59.638009 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:45:59.638011 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:45:59.638067 (+ 56us) tablet.cc:699] Row locks acquired
0921 18:45:59.638068 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:45:59.638106 (+ 38us) write_op.cc:270] Start()
0921 18:46:00.628238 (+990132us) write_op.cc:276] Timestamp: P: 1790016359638067 usec, L: 0
0921 18:46:00.628240 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:46:00.628417 (+ 177us) log.cc:842] Serialized 354 byte log entry
0921 18:46:01.044624 (+416207us) op_driver.cc:444] REPLICATION: finished
0921 18:46:01.044955 (+ 331us) write_op.cc:301] APPLY: starting
0921 18:46:01.044971 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:01.044999 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:01.045343 (+ 344us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:01.045402 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:01.045426 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:01.045431 (+ 5us) write_op.cc:312] APPLY: finished
0921 18:46:01.045519 (+ 88us) log.cc:842] Serialized 43 byte log entry
0921 18:46:01.045570 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:01.045580 (+ 10us) write_op.cc:454] Released schema lock
0921 18:46:01.045585 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":282,"num_ops":1,"prepare.queue_time_us":254119,"prepare.run_cpu_time_us":644,"prepare.run_wall_time_us":990717,"raft.queue_time_us":40,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"replication_time_us":416356,"thread_start_us":82,"threads_started":1}]]}
I20260921 18:46:01.504951 7037 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:01.505100 7037 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:01.505669 7037 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:46:01.506022 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 49 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:01.506181 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 49 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:46:01.506578 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:01.616250 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:01.617748 7056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:42348 (ReqId={client: 4d673e529d214ff1a6d96d962d8a8681, seq_no=47, attempt_no=0}) took 1483 ms. Trace:
I20260921 18:46:01.617843 7056 rpcz_store.cc:276] 0921 18:46:00.134255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:00.134330 (+ 75us) service_pool.cc:227] Handling call
0921 18:46:01.617733 (+1483403us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:00.628557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 1b4ae6eac23448b79a40085d1e15c4d3
0921 18:46:00.628648 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:00.628652 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:46:00.628669 (+ 17us) tablet.cc:660] Decoding operations
0921 18:46:00.628743 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:00.628752 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:00.628755 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:46:00.628798 (+ 43us) tablet.cc:699] Row locks acquired
0921 18:46:00.628800 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:46:00.628837 (+ 37us) write_op.cc:270] Start()
0921 18:46:00.930986 (+302149us) write_op.cc:276] Timestamp: P: 1790016360628811 usec, L: 0
0921 18:46:00.930990 (+ 4us) op_driver.cc:331] REPLICATION: starting
0921 18:46:00.931175 (+ 185us) log.cc:842] Serialized 354 byte log entry
0921 18:46:01.617145 (+685970us) op_driver.cc:444] REPLICATION: finished
0921 18:46:01.617345 (+ 200us) write_op.cc:301] APPLY: starting
0921 18:46:01.617357 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:01.617380 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:01.617393 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:01.617441 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:01.617459 (+ 18us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:01.617461 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:46:01.617568 (+ 107us) log.cc:842] Serialized 43 byte log entry
0921 18:46:01.617638 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:01.617648 (+ 10us) write_op.cc:454] Released schema lock
0921 18:46:01.617653 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":146,"num_ops":1,"prepare.queue_time_us":493893,"prepare.run_cpu_time_us":604,"prepare.run_wall_time_us":302673,"replication_time_us":686133,"thread_start_us":63,"threads_started":1}]]}
I20260921 18:46:01.871356 6974 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:01.871515 6974 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:01.872146 6974 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457), dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181)
I20260921 18:46:01.872643 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 51 } } ignore_live_leader: false dest_uuid: "3ed466b1d5d14005adcb29914704ab7c" is_pre_election: true
I20260921 18:46:01.872656 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "7a46f3a6b1634626be69e0782c521f15" candidate_term: 3 candidate_status { last_received { term: 2 index: 51 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:01.873086 6536 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a46f3a6b1634626be69e0782c521f15; no voters: 3ed466b1d5d14005adcb29914704ab7c, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:01.873297 6974 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 7a46f3a6b1634626be69e0782c521f15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:01.933431 6582 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:01.935359 6713 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:46:02.163491 7037 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260921 18:46:02.165818 6451 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:02.422528 7037 raft_consensus.cc:482] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3f9f8b70dc432699d59db2668757d9)
I20260921 18:46:02.422673 7037 raft_consensus.cc:504] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed466b1d5d14005adcb29914704ab7c" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 38457 } } peers { permanent_uuid: "dd3f9f8b70dc432699d59db2668757d9" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36181 } } peers { permanent_uuid: "7a46f3a6b1634626be69e0782c521f15" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 36893 } }
I20260921 18:46:02.423164 7037 leader_election.cc:288] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dd3f9f8b70dc432699d59db2668757d9 (127.3.36.131:36181), 7a46f3a6b1634626be69e0782c521f15 (127.3.36.130:36893)
I20260921 18:46:02.423496 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 51 } } ignore_live_leader: false dest_uuid: "dd3f9f8b70dc432699d59db2668757d9" is_pre_election: true
I20260921 18:46:02.427977 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4ae6eac23448b79a40085d1e15c4d3" candidate_uuid: "3ed466b1d5d14005adcb29914704ab7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 51 } } ignore_live_leader: false dest_uuid: "7a46f3a6b1634626be69e0782c521f15" is_pre_election: true
I20260921 18:46:02.428318 6404 leader_election.cc:302] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ed466b1d5d14005adcb29914704ab7c; no voters: 7a46f3a6b1634626be69e0782c521f15, dd3f9f8b70dc432699d59db2668757d9
I20260921 18:46:02.428525 7037 raft_consensus.cc:2726] T 1b4ae6eac23448b79a40085d1e15c4d3 P 3ed466b1d5d14005adcb29914704ab7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
aff9f6524d084d6e8e8151b0b2225e35 | 127.3.36.190:46581 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:39483 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3ed466b1d5d14005adcb29914704ab7c | 127.3.36.129:38457 | HEALTHY | <none> | 0 | 0
7a46f3a6b1634626be69e0782c521f15 | 127.3.36.130:36893 | HEALTHY | <none> | 0 | 0
dd3f9f8b70dc432699d59db2668757d9 | 127.3.36.131:36181 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:38457
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:36893
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:36181
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:38457
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:36893
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:36181
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:39483 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260921 18:46:02.445241 3218 log_verifier.cc:126] Checking tablet 1b4ae6eac23448b79a40085d1e15c4d3
I20260921 18:46:02.486657 3218 log_verifier.cc:177] Verified matching terms for 51 ops in tablet 1b4ae6eac23448b79a40085d1e15c4d3
I20260921 18:46:02.510906 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 6389
W20260921 18:46:02.517557 6667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.36.129:38457 (error 108) [suppressed 1 similar messages]
I20260921 18:46:02.517901 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 6520
W20260921 18:46:02.519198 6667 consensus_peers.cc:487] T 1b4ae6eac23448b79a40085d1e15c4d3 P dd3f9f8b70dc432699d59db2668757d9 -> Peer 3ed466b1d5d14005adcb29914704ab7c (127.3.36.129:38457): Couldn't send request to peer 3ed466b1d5d14005adcb29914704ab7c. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.129:38457: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260921 18:46:02.524662 6666 connection.cc:535] client connection to 127.3.36.130:36893 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:46:02.524945 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 6651
I20260921 18:46:02.529433 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 6298
2026-09-21T18:46:02Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (32316 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20260921 18:46:02.552901 3218 test_util.cc:268] Using random seed: -1452368319
I20260921 18:46:02.553673 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:02.553719 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:02.553732 3218 ts_itest-base.cc:114] 5 tablet servers
I20260921 18:46:02.553743 3218 ts_itest-base.cc:115] 5 replicas per TS
I20260921 18:46:02.553753 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:02Z Disabled control of system clock
I20260921 18:46:02.561839 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:42057
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:42057 with env {}
W20260921 18:46:02.666074 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:02.666356 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:02.666426 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:02.670368 7106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:02.670449 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:02.670482 7106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:02.670539 7106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:02.675058 7106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:42057
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:42057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:02.676031 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:02.677309 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260921 18:46:02.685372 7106 server_base.cc:1034] running on GCE node
W20260921 18:46:02.685393 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:02.685330 7112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:02.685330 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:02.686053 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:02.686985 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:02.688138 7106 hybrid_clock.cc:648] HybridClock initialized: now 1790016362688108 us; error 43 us; skew 500 ppm
I20260921 18:46:02.689898 7106 webserver.cc:460] Webserver started at http://127.3.36.190:46589/ using document root <none> and password file <none>
I20260921 18:46:02.690423 7106 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:02.690482 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:02.690723 7106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:02.692489 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "249245e0706c462993d0cb93e01237a9"
format_stamp: "Formatted at 2026-09-21 18:46:02 on dist-test-slave-l4st"
I20260921 18:46:02.693028 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "249245e0706c462993d0cb93e01237a9"
format_stamp: "Formatted at 2026-09-21 18:46:02 on dist-test-slave-l4st"
I20260921 18:46:02.696550 7106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260921 18:46:02.698788 7120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:02.699859 7106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:02.699973 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "249245e0706c462993d0cb93e01237a9"
format_stamp: "Formatted at 2026-09-21 18:46:02 on dist-test-slave-l4st"
I20260921 18:46:02.700065 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:02.713155 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:02.713749 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:02.713934 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:02.721024 7106 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:42057
I20260921 18:46:02.721046 7172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:42057 every 8 connection(s)
I20260921 18:46:02.722005 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:02.725028 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:02.726447 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 7106
I20260921 18:46:02.726570 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:02.731606 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9: Bootstrap starting.
I20260921 18:46:02.734030 7173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:02.734887 7173 log.cc:824] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:02.736833 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9: No bootstrap required, opened a new log
I20260921 18:46:02.740099 7173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249245e0706c462993d0cb93e01237a9" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42057 } }
I20260921 18:46:02.740291 7173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:02.740365 7173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249245e0706c462993d0cb93e01237a9, State: Initialized, Role: FOLLOWER
I20260921 18:46:02.740751 7173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249245e0706c462993d0cb93e01237a9" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42057 } }
I20260921 18:46:02.740885 7173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:02.740945 7173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:02.741068 7173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:02.741827 7173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249245e0706c462993d0cb93e01237a9" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42057 } }
I20260921 18:46:02.742146 7173 leader_election.cc:302] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249245e0706c462993d0cb93e01237a9; no voters:
I20260921 18:46:02.742413 7173 leader_election.cc:288] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:02.742493 7178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:02.742761 7178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [term 1 LEADER]: Becoming Leader. State: Replica: 249245e0706c462993d0cb93e01237a9, State: Running, Role: LEADER
I20260921 18:46:02.743067 7178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249245e0706c462993d0cb93e01237a9" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42057 } }
I20260921 18:46:02.743496 7173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:02.744588 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249245e0706c462993d0cb93e01237a9. Latest consensus state: current_term: 1 leader_uuid: "249245e0706c462993d0cb93e01237a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249245e0706c462993d0cb93e01237a9" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42057 } } }
I20260921 18:46:02.744709 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:02.745072 7187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:02.745044 7179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249245e0706c462993d0cb93e01237a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249245e0706c462993d0cb93e01237a9" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 42057 } } }
I20260921 18:46:02.745700 7179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:02.747864 7187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:02.753455 7187 catalog_manager.cc:1347] Generated new cluster ID: 541f728eca7b44d5ae0064d88a846f11
I20260921 18:46:02.753562 7187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:02.773532 7187 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:02.774469 7187 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:02.780882 7187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249245e0706c462993d0cb93e01237a9: Generated new TSK 0
I20260921 18:46:02.781569 7187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:02.785306 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260921 18:46:02.892778 7197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:02.893002 7197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:02.893034 7197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:02.897020 7197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:02.897100 7197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:02.901721 7197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.7197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:02.902606 7197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:02.903937 7197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:02.912815 7205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:02.912829 7203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:02.912832 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:02.913054 7197 server_base.cc:1034] running on GCE node
I20260921 18:46:02.913580 7197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:02.914122 7197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:02.915277 7197 hybrid_clock.cc:648] HybridClock initialized: now 1790016362915264 us; error 24 us; skew 500 ppm
I20260921 18:46:02.916992 7197 webserver.cc:460] Webserver started at http://127.3.36.129:43939/ using document root <none> and password file <none>
I20260921 18:46:02.917599 7197 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:02.917670 7197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:02.917897 7197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:02.919672 7197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
format_stamp: "Formatted at 2026-09-21 18:46:02 on dist-test-slave-l4st"
I20260921 18:46:02.920199 7197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
format_stamp: "Formatted at 2026-09-21 18:46:02 on dist-test-slave-l4st"
I20260921 18:46:02.923902 7197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260921 18:46:02.926072 7211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:02.927183 7197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:02.927294 7197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
format_stamp: "Formatted at 2026-09-21 18:46:02 on dist-test-slave-l4st"
I20260921 18:46:02.927383 7197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:02.941015 7197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:02.941702 7197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:02.941880 7197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:02.942412 7197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:02.943259 7197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:02.943303 7197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:02.943368 7197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:02.943398 7197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:02.951562 7197 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:32775
I20260921 18:46:02.951627 7324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:32775 every 8 connection(s)
I20260921 18:46:02.952414 7197 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:02.961032 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 7197
I20260921 18:46:02.961153 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:02.961964 7325 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:02.962172 7325 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:02.962694 7325 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:02.963488 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260921 18:46:02.964133 7137 ts_manager.cc:194] Registered new tserver with Master: ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775)
I20260921 18:46:02.965312 7137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:42613
W20260921 18:46:03.068014 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:03.068282 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:03.068348 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:03.072268 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:03.072402 7328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:03.076902 7328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.7328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:03.077885 7328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:03.079171 7328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:03.087778 7333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.087842 7334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.087786 7336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:03.088187 7328 server_base.cc:1034] running on GCE node
I20260921 18:46:03.088603 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:03.089181 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:03.090365 7328 hybrid_clock.cc:648] HybridClock initialized: now 1790016363090355 us; error 31 us; skew 500 ppm
I20260921 18:46:03.092164 7328 webserver.cc:460] Webserver started at http://127.3.36.130:41819/ using document root <none> and password file <none>
I20260921 18:46:03.092707 7328 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:03.092794 7328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:03.092995 7328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:03.094908 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f367cedc80014733882b43e8a3c87d45"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.095456 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f367cedc80014733882b43e8a3c87d45"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.098850 7328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:03.100930 7342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.102108 7328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:03.102229 7328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f367cedc80014733882b43e8a3c87d45"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.102329 7328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:03.116935 7328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:03.117568 7328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:03.117749 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:03.118268 7328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:03.119167 7328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:03.119235 7328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.119302 7328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:03.119354 7328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.127482 7328 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:42667
I20260921 18:46:03.127509 7455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:42667 every 8 connection(s)
I20260921 18:46:03.128296 7328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:03.129158 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 7328
I20260921 18:46:03.129269 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:03.131486 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260921 18:46:03.139099 7456 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:03.139328 7456 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:03.139833 7456 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:03.140805 7137 ts_manager.cc:194] Registered new tserver with Master: f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667)
I20260921 18:46:03.141335 7137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:57831
W20260921 18:46:03.239861 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:03.240099 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:03.240168 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:03.244185 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:03.244288 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:03.249115 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.7459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:03.250089 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:03.251380 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:03.260428 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.260401 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.260413 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:03.260565 7459 server_base.cc:1034] running on GCE node
I20260921 18:46:03.260980 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:03.261655 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:03.262816 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790016363262801 us; error 23 us; skew 500 ppm
I20260921 18:46:03.264647 7459 webserver.cc:460] Webserver started at http://127.3.36.131:42877/ using document root <none> and password file <none>
I20260921 18:46:03.265192 7459 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:03.265251 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:03.265527 7459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:03.267310 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.267853 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.271317 7459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:03.273416 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.274564 7459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:03.274690 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.274801 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:03.285403 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:03.285980 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:03.286178 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:03.286712 7459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:03.287545 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:03.287622 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.287688 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:03.287741 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.295718 7459 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:37089
I20260921 18:46:03.295742 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:37089 every 8 connection(s)
I20260921 18:46:03.296500 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:03.297075 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 7459
I20260921 18:46:03.297199 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:03.299530 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.132:0
--local_ip_for_outbound_sockets=127.3.36.132
--webserver_interface=127.3.36.132
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260921 18:46:03.307374 7587 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:03.307621 7587 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:03.308252 7587 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:03.309314 7137 ts_manager.cc:194] Registered new tserver with Master: 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089)
I20260921 18:46:03.310009 7137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:58775
W20260921 18:46:03.408123 7590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:03.408362 7590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:03.408429 7590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:03.412379 7590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:03.412495 7590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.132
I20260921 18:46:03.417292 7590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.36.132
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.7590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.132
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:03.418298 7590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:03.419607 7590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:03.428759 7598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:03.428807 7590 server_base.cc:1034] running on GCE node
W20260921 18:46:03.428771 7595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.428759 7596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:03.429319 7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:03.429970 7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:03.431152 7590 hybrid_clock.cc:648] HybridClock initialized: now 1790016363431140 us; error 36 us; skew 500 ppm
I20260921 18:46:03.432799 7590 webserver.cc:460] Webserver started at http://127.3.36.132:33247/ using document root <none> and password file <none>
I20260921 18:46:03.433326 7590 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:03.433411 7590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:03.433660 7590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:03.435423 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "af7b9688bd714366830a341b43b5bab9"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.435954 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "af7b9688bd714366830a341b43b5bab9"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.439386 7590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:03.441540 7604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.442610 7590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:03.442746 7590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "af7b9688bd714366830a341b43b5bab9"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.442843 7590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:03.465813 7590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:03.466429 7590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:03.466614 7590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:03.467199 7590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:03.468084 7590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:03.468158 7590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.468227 7590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:03.468266 7590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.476334 7590 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.132:34763
I20260921 18:46:03.476357 7717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.132:34763 every 8 connection(s)
I20260921 18:46:03.477177 7590 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260921 18:46:03.485834 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 7590
I20260921 18:46:03.485961 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260921 18:46:03.487093 7718 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:03.487305 7718 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:03.487766 7718 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:03.488070 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.133:0
--local_ip_for_outbound_sockets=127.3.36.133
--webserver_interface=127.3.36.133
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260921 18:46:03.488786 7137 ts_manager.cc:194] Registered new tserver with Master: af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:03.489316 7137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.132:56435
W20260921 18:46:03.596314 7721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:03.596531 7721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:03.596565 7721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:03.600488 7721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:03.600569 7721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.133
I20260921 18:46:03.605238 7721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.3.36.133
--webserver_port=0
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.7721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.133
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:03.606223 7721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:03.607600 7721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:03.616338 7729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.616338 7727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:03.616362 7726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:03.616525 7721 server_base.cc:1034] running on GCE node
I20260921 18:46:03.617008 7721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:03.617646 7721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:03.618832 7721 hybrid_clock.cc:648] HybridClock initialized: now 1790016363618793 us; error 50 us; skew 500 ppm
I20260921 18:46:03.620486 7721 webserver.cc:460] Webserver started at http://127.3.36.133:36867/ using document root <none> and password file <none>
I20260921 18:46:03.621007 7721 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:03.621093 7721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:03.621313 7721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:03.623086 7721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7a966bfd1b9241fc807e119bbf645861"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.623668 7721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7a966bfd1b9241fc807e119bbf645861"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.627199 7721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:03.629261 7735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.630451 7721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:03.630559 7721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7a966bfd1b9241fc807e119bbf645861"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:03.630656 7721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:03.640511 7721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:03.641052 7721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:03.641242 7721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:03.641801 7721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:03.642618 7721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:03.642665 7721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.642727 7721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:03.642765 7721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:03.651435 7721 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.133:35431
I20260921 18:46:03.651456 7848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.133:35431 every 8 connection(s)
I20260921 18:46:03.652249 7721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260921 18:46:03.653787 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 7721
I20260921 18:46:03.653904 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260921 18:46:03.661396 7849 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:03.661624 7849 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:03.662024 7849 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:03.662876 7137 ts_manager.cc:194] Registered new tserver with Master: 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431)
I20260921 18:46:03.663372 7137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.133:42271
I20260921 18:46:03.667716 3218 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260921 18:46:03.676944 7137 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42648:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260921 18:46:03.680039 7137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260921 18:46:03.697368 7390 tablet_service.cc:1467] Processing CreateTablet for tablet d977daa5e3514a828d2098599ed2b135 (DEFAULT_TABLE table=TestTable [id=5a9e88d92840492c9f0df8f4cc52f5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:03.697731 7521 tablet_service.cc:1467] Processing CreateTablet for tablet d977daa5e3514a828d2098599ed2b135 (DEFAULT_TABLE table=TestTable [id=5a9e88d92840492c9f0df8f4cc52f5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:03.698443 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d977daa5e3514a828d2098599ed2b135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:03.698652 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d977daa5e3514a828d2098599ed2b135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:03.698585 7783 tablet_service.cc:1467] Processing CreateTablet for tablet d977daa5e3514a828d2098599ed2b135 (DEFAULT_TABLE table=TestTable [id=5a9e88d92840492c9f0df8f4cc52f5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:03.699448 7783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d977daa5e3514a828d2098599ed2b135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:03.702515 7652 tablet_service.cc:1467] Processing CreateTablet for tablet d977daa5e3514a828d2098599ed2b135 (DEFAULT_TABLE table=TestTable [id=5a9e88d92840492c9f0df8f4cc52f5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:03.703409 7652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d977daa5e3514a828d2098599ed2b135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:03.706336 7870 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: Bootstrap starting.
I20260921 18:46:03.706518 7871 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: Bootstrap starting.
I20260921 18:46:03.708495 7871 tablet_bootstrap.cc:654] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:03.708529 7870 tablet_bootstrap.cc:654] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:03.709210 7871 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:03.709280 7870 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:03.709589 7873 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Bootstrap starting.
I20260921 18:46:03.709875 7872 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Bootstrap starting.
I20260921 18:46:03.711086 7870 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: No bootstrap required, opened a new log
I20260921 18:46:03.711269 7870 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:03.711884 7872 tablet_bootstrap.cc:654] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:03.711999 7873 tablet_bootstrap.cc:654] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:03.712621 7872 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:03.712752 7873 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:03.713569 7259 tablet_service.cc:1467] Processing CreateTablet for tablet d977daa5e3514a828d2098599ed2b135 (DEFAULT_TABLE table=TestTable [id=5a9e88d92840492c9f0df8f4cc52f5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:03.714484 7872 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: No bootstrap required, opened a new log
I20260921 18:46:03.714484 7871 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: No bootstrap required, opened a new log
I20260921 18:46:03.714485 7259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d977daa5e3514a828d2098599ed2b135. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:03.714666 7872 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:46:03.714687 7871 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260921 18:46:03.715416 7870 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.715694 7870 raft_consensus.cc:374] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:03.715755 7870 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef8886ab4d4417c8b0c251b1105d6ba, State: Initialized, Role: FOLLOWER
I20260921 18:46:03.716235 7870 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.717203 7870 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260921 18:46:03.717545 7587 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
I20260921 18:46:03.718950 7872 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.719192 7872 raft_consensus.cc:374] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:03.719245 7872 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a966bfd1b9241fc807e119bbf645861, State: Initialized, Role: FOLLOWER
I20260921 18:46:03.719436 7871 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.719714 7871 raft_consensus.cc:374] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:03.719691 7872 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.719801 7871 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f367cedc80014733882b43e8a3c87d45, State: Initialized, Role: FOLLOWER
I20260921 18:46:03.720316 7871 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.721444 7871 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260921 18:46:03.721637 7872 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260921 18:46:03.721838 7456 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
I20260921 18:46:03.721897 7849 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
I20260921 18:46:03.724937 7881 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3: Bootstrap starting.
I20260921 18:46:03.725899 7873 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: No bootstrap required, opened a new log
I20260921 18:46:03.726104 7873 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20260921 18:46:03.726955 7881 tablet_bootstrap.cc:654] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:03.727658 7881 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:03.729391 7881 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3: No bootstrap required, opened a new log
I20260921 18:46:03.729575 7881 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:03.729761 7873 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.729987 7873 raft_consensus.cc:374] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:03.730046 7873 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7b9688bd714366830a341b43b5bab9, State: Initialized, Role: FOLLOWER
I20260921 18:46:03.730461 7873 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.731029 7718 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
I20260921 18:46:03.731330 7873 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:46:03.733803 7881 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.734030 7881 raft_consensus.cc:374] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:03.734072 7881 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8442fea5d54cff8fccdb7b5d13fdb3, State: Initialized, Role: FOLLOWER
I20260921 18:46:03.734426 7881 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.735018 7325 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
I20260921 18:46:03.735262 7881 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260921 18:46:03.797672 7588 tablet.cc:2369] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:03.842000 7883 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:46:03.842204 7883 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.843755 7883 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089)
I20260921 18:46:03.848905 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:03.849253 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 0.
I20260921 18:46:03.849679 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:03.849953 7279 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 0.
I20260921 18:46:03.850000 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" is_pre_election: true
I20260921 18:46:03.850303 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 0.
I20260921 18:46:03.850358 7605 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: ae8442fea5d54cff8fccdb7b5d13fdb3, af7b9688bd714366830a341b43b5bab9, f367cedc80014733882b43e8a3c87d45; no voters:
I20260921 18:46:03.850492 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a966bfd1b9241fc807e119bbf645861" is_pre_election: true
I20260921 18:46:03.850723 7883 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:46:03.850766 7803 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 0.
I20260921 18:46:03.850821 7883 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:46:03.850889 7883 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:03.851861 7883 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.852330 7883 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [CANDIDATE]: Term 1 election: Requested vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089)
I20260921 18:46:03.852608 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
I20260921 18:46:03.852624 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a966bfd1b9241fc807e119bbf645861"
I20260921 18:46:03.852653 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45"
I20260921 18:46:03.852684 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "af7b9688bd714366830a341b43b5bab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
I20260921 18:46:03.852747 7803 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:03.852752 7410 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:03.852741 7279 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:03.852782 7541 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:03.853878 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 1.
I20260921 18:46:03.853878 7803 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 1.
I20260921 18:46:03.854012 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 1.
I20260921 18:46:03.854125 7279 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7b9688bd714366830a341b43b5bab9 in term 1.
I20260921 18:46:03.854197 7606 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, 7a966bfd1b9241fc807e119bbf645861, af7b9688bd714366830a341b43b5bab9; no voters:
I20260921 18:46:03.854349 7883 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:03.854543 7883 raft_consensus.cc:686] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 1 LEADER]: Becoming Leader. State: Replica: af7b9688bd714366830a341b43b5bab9, State: Running, Role: LEADER
I20260921 18:46:03.854878 7883 consensus_queue.cc:237] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:03.858105 7135 catalog_manager.cc:5495] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to af7b9688bd714366830a341b43b5bab9 (127.3.36.132). New cstate: current_term: 1 leader_uuid: "af7b9688bd714366830a341b43b5bab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } health_report { overall_health: HEALTHY } } }
W20260921 18:46:03.879640 7457 tablet.cc:2369] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:03.883272 3218 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260921 18:46:03.884567 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ae8442fea5d54cff8fccdb7b5d13fdb3 to finish bootstrapping
I20260921 18:46:03.890511 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f367cedc80014733882b43e8a3c87d45 to finish bootstrapping
I20260921 18:46:03.895641 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5ef8886ab4d4417c8b0c251b1105d6ba to finish bootstrapping
I20260921 18:46:03.900380 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver af7b9688bd714366830a341b43b5bab9 to finish bootstrapping
W20260921 18:46:03.903106 7850 tablet.cc:2369] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:03.905323 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7a966bfd1b9241fc807e119bbf645861 to finish bootstrapping
I20260921 18:46:03.918144 3218 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20260921 18:46:03.931869 7541 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Refusing update from remote peer af7b9688bd714366830a341b43b5bab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:03.931869 7410 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Refusing update from remote peer af7b9688bd714366830a341b43b5bab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:03.931869 7279 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Refusing update from remote peer af7b9688bd714366830a341b43b5bab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:03.931869 7803 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Refusing update from remote peer af7b9688bd714366830a341b43b5bab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:03.932631 7895 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:03.932821 7894 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:03.932978 7883 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:03.933147 7888 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:03.941175 7893 mvcc.cc:205] Tried to move back new op lower bound from 7331907026657497088 to 7331907026352795648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907026657497088}]
I20260921 18:46:03.947862 7898 mvcc.cc:205] Tried to move back new op lower bound from 7331907026657497088 to 7331907026352795648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907026657497088}]
I20260921 18:46:03.948040 7896 mvcc.cc:205] Tried to move back new op lower bound from 7331907026657497088 to 7331907026352795648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907026657497088}]
I20260921 18:46:03.948624 7897 mvcc.cc:205] Tried to move back new op lower bound from 7331907026657497088 to 7331907026352795648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907026657497088}]
I20260921 18:46:03.970672 3218 raft_consensus_election-itest.cc:408] Killing current leader af7b9688bd714366830a341b43b5bab9...
I20260921 18:46:03.970768 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 7590
W20260921 18:46:03.977330 7215 connection.cc:535] server connection from 127.3.36.132:35071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:46:03.977686 3218 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260921 18:46:03.979354 3218 test_util.cc:268] Using random seed: -1450941867
I20260921 18:46:04.099187 3218 test_util.cc:268] Using random seed: -1450822040
I20260921 18:46:04.305402 3218 test_util.cc:268] Using random seed: -1450615822
I20260921 18:46:04.611387 3218 test_util.cc:268] Using random seed: -1450309838
I20260921 18:46:05.017503 3218 test_util.cc:268] Using random seed: -1449903755
I20260921 18:46:05.469961 7912 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af7b9688bd714366830a341b43b5bab9)
I20260921 18:46:05.470115 7912 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:05.470357 7913 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af7b9688bd714366830a341b43b5bab9)
I20260921 18:46:05.470485 7913 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:05.470907 7914 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af7b9688bd714366830a341b43b5bab9)
I20260921 18:46:05.471021 7914 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:05.472316 7912 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:05.472518 7919 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader af7b9688bd714366830a341b43b5bab9)
I20260921 18:46:05.472640 7919 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:05.474287 7913 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:05.474643 7914 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:05.476158 7919 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
W20260921 18:46:05.477950 7475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.478783 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "7a966bfd1b9241fc807e119bbf645861" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:05.478921 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a966bfd1b9241fc807e119bbf645861 in term 1.
W20260921 18:46:05.480028 7737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
W20260921 18:46:05.480242 7344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.480988 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "7a966bfd1b9241fc807e119bbf645861" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" is_pre_election: true
I20260921 18:46:05.481117 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a966bfd1b9241fc807e119bbf645861 in term 1.
W20260921 18:46:05.481372 7213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.481429 7737 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, 7a966bfd1b9241fc807e119bbf645861, f367cedc80014733882b43e8a3c87d45; no voters:
I20260921 18:46:05.481762 7912 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260921 18:46:05.481827 7912 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Starting leader election (detected failure of leader af7b9688bd714366830a341b43b5bab9)
I20260921 18:46:05.481895 7912 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:05.483098 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:05.483227 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 1.
I20260921 18:46:05.484716 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" is_pre_election: true
I20260921 18:46:05.484843 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in term 1.
W20260921 18:46:05.485641 7344 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.485733 7912 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:05.486259 7912 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [CANDIDATE]: Term 2 election: Requested vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:05.486634 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "7a966bfd1b9241fc807e119bbf645861" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45"
I20260921 18:46:05.486754 7410 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:05.486999 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "7a966bfd1b9241fc807e119bbf645861" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
I20260921 18:46:05.487104 7541 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:05.487870 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a966bfd1b9241fc807e119bbf645861 in term 2.
I20260921 18:46:05.487871 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" is_pre_election: true
I20260921 18:46:05.488225 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a966bfd1b9241fc807e119bbf645861 in term 2.
I20260921 18:46:05.489025 7737 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, 7a966bfd1b9241fc807e119bbf645861, f367cedc80014733882b43e8a3c87d45; no voters:
W20260921 18:46:05.489228 7213 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
W20260921 18:46:05.489423 7475 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.489651 7912 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 2 FOLLOWER]: Leader election won for term 2
W20260921 18:46:05.490082 7737 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.490475 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a966bfd1b9241fc807e119bbf645861" is_pre_election: true
I20260921 18:46:05.490770 7912 raft_consensus.cc:686] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 2 LEADER]: Becoming Leader. State: Replica: 7a966bfd1b9241fc807e119bbf645861, State: Running, Role: LEADER
I20260921 18:46:05.491110 7912 consensus_queue.cc:237] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:05.492161 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "7a966bfd1b9241fc807e119bbf645861" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:05.492223 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "7a966bfd1b9241fc807e119bbf645861" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
I20260921 18:46:05.492280 7279 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a966bfd1b9241fc807e119bbf645861 in term 1.
I20260921 18:46:05.493659 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:05.493841 7410 raft_consensus.cc:2379] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in current term 2: Already voted for candidate 7a966bfd1b9241fc807e119bbf645861 in this term.
I20260921 18:46:05.494068 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:05.494175 7278 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 1.
I20260921 18:46:05.494450 7474 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters: af7b9688bd714366830a341b43b5bab9
I20260921 18:46:05.494699 7913 raft_consensus.cc:2741] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260921 18:46:05.495625 7214 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3; no voters: 7a966bfd1b9241fc807e119bbf645861, af7b9688bd714366830a341b43b5bab9, f367cedc80014733882b43e8a3c87d45
I20260921 18:46:05.495693 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:05.495777 7278 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f367cedc80014733882b43e8a3c87d45 in term 1.
W20260921 18:46:05.496449 7737 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:05.496491 7919 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:05.496537 7136 catalog_manager.cc:5495] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 reported cstate change: term changed from 1 to 2, leader changed from af7b9688bd714366830a341b43b5bab9 (127.3.36.132) to 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133). New cstate: current_term: 2 leader_uuid: "7a966bfd1b9241fc807e119bbf645861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:05.496830 7803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a966bfd1b9241fc807e119bbf645861" is_pre_election: true
I20260921 18:46:05.497059 7802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7a966bfd1b9241fc807e119bbf645861" is_pre_election: true
I20260921 18:46:05.497180 7345 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters: 5ef8886ab4d4417c8b0c251b1105d6ba, 7a966bfd1b9241fc807e119bbf645861, af7b9688bd714366830a341b43b5bab9
I20260921 18:46:05.497447 7914 raft_consensus.cc:2726] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:46:05.497856 7919 raft_consensus.cc:2726] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:46:05.523629 3218 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20260921 18:46:05.525985 7853 meta_cache.cc:302] tablet d977daa5e3514a828d2098599ed2b135: replica af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763) has failed: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
W20260921 18:46:05.534547 7370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32960: Illegal state: replica f367cedc80014733882b43e8a3c87d45 is not leader of this config: current role FOLLOWER
W20260921 18:46:05.542024 7239 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44584: Illegal state: replica ae8442fea5d54cff8fccdb7b5d13fdb3 is not leader of this config: current role FOLLOWER
I20260921 18:46:05.553381 7278 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Refusing update from remote peer 7a966bfd1b9241fc807e119bbf645861: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260921 18:46:05.553850 7410 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Refusing update from remote peer 7a966bfd1b9241fc807e119bbf645861: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20260921 18:46:05.553718 7541 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Refusing update from remote peer 7a966bfd1b9241fc807e119bbf645861: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20260921 18:46:05.554392 7737 consensus_peers.cc:487] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 -> Peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Couldn't send request to peer af7b9688bd714366830a341b43b5bab9. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260921 18:46:05.554503 7958 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260921 18:46:05.554670 7912 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260921 18:46:05.555219 7957 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260921 18:46:05.581667 3218 raft_consensus_election-itest.cc:408] Killing current leader 7a966bfd1b9241fc807e119bbf645861...
I20260921 18:46:05.581791 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 7721
W20260921 18:46:05.587985 7214 connection.cc:535] server connection from 127.3.36.133:59465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260921 18:46:05.588002 7475 connection.cc:535] server connection from 127.3.36.133:40947 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:46:05.588466 3218 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20260921 18:46:05.590086 3218 test_util.cc:268] Using random seed: -1449331135
I20260921 18:46:05.695492 3218 test_util.cc:268] Using random seed: -1449225734
I20260921 18:46:05.901250 3218 test_util.cc:268] Using random seed: -1449019976
I20260921 18:46:06.207036 3218 test_util.cc:268] Using random seed: -1448714189
I20260921 18:46:06.612533 3218 test_util.cc:268] Using random seed: -1448308693
I20260921 18:46:07.081907 7971 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a966bfd1b9241fc807e119bbf645861)
I20260921 18:46:07.082082 7971 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:07.082285 7972 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a966bfd1b9241fc807e119bbf645861)
I20260921 18:46:07.082314 7973 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7a966bfd1b9241fc807e119bbf645861)
I20260921 18:46:07.082422 7973 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:07.082422 7972 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:07.082959 7971 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:07.083145 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:07.083272 7278 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f367cedc80014733882b43e8a3c87d45 in term 2.
I20260921 18:46:07.083380 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:07.083513 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 2.
I20260921 18:46:07.083601 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" is_pre_election: true
I20260921 18:46:07.083727 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f367cedc80014733882b43e8a3c87d45 in term 2.
I20260921 18:46:07.083940 7972 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:07.084276 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:07.084383 7278 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 2.
I20260921 18:46:07.084457 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:07.084591 7410 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in term 2.
I20260921 18:46:07.084753 7973 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:07.084908 7344 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters:
I20260921 18:46:07.085285 7971 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260921 18:46:07.085345 7971 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a966bfd1b9241fc807e119bbf645861)
I20260921 18:46:07.085384 7971 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 2 FOLLOWER]: Advancing to term 3
I20260921 18:46:07.084790 7474 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters:
I20260921 18:46:07.085523 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" is_pre_election: true
I20260921 18:46:07.085640 7541 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in term 2.
I20260921 18:46:07.085717 7973 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260921 18:46:07.085768 7973 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a966bfd1b9241fc807e119bbf645861)
I20260921 18:46:07.085803 7973 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 2 FOLLOWER]: Advancing to term 3
W20260921 18:46:07.085913 7345 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
I20260921 18:46:07.085912 7213 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters:
I20260921 18:46:07.086090 7972 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260921 18:46:07.086138 7972 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7a966bfd1b9241fc807e119bbf645861)
I20260921 18:46:07.086177 7972 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 2 FOLLOWER]: Advancing to term 3
I20260921 18:46:07.086591 7971 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
W20260921 18:46:07.086705 7214 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
W20260921 18:46:07.086812 7213 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:07.086858 7973 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:07.087052 7971 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 election: Requested vote from peers ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:07.087285 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
I20260921 18:46:07.087365 7973 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 election: Requested vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:07.087718 7279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
W20260921 18:46:07.087852 7344 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:07.087896 7972 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
W20260921 18:46:07.088085 7476 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
I20260921 18:46:07.088227 7278 raft_consensus.cc:2379] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f367cedc80014733882b43e8a3c87d45 in current term 3: Already voted for candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in this term.
I20260921 18:46:07.088385 7972 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 election: Requested vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:07.088423 7410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45"
I20260921 18:46:07.088553 7410 raft_consensus.cc:2379] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ef8886ab4d4417c8b0c251b1105d6ba in current term 3: Already voted for candidate f367cedc80014733882b43e8a3c87d45 in this term.
I20260921 18:46:07.088687 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45"
I20260921 18:46:07.088797 7409 raft_consensus.cc:2379] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in current term 3: Already voted for candidate f367cedc80014733882b43e8a3c87d45 in this term.
I20260921 18:46:07.088982 7540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
I20260921 18:46:07.089148 7540 raft_consensus.cc:2379] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ae8442fea5d54cff8fccdb7b5d13fdb3 in current term 3: Already voted for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in this term.
I20260921 18:46:07.089247 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "f367cedc80014733882b43e8a3c87d45" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba"
I20260921 18:46:07.089370 7541 raft_consensus.cc:2379] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f367cedc80014733882b43e8a3c87d45 in current term 3: Already voted for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in this term.
W20260921 18:46:07.089416 7345 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
I20260921 18:46:07.089635 7344 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: f367cedc80014733882b43e8a3c87d45; no voters: 5ef8886ab4d4417c8b0c251b1105d6ba, 7a966bfd1b9241fc807e119bbf645861, ae8442fea5d54cff8fccdb7b5d13fdb3
I20260921 18:46:07.089769 7971 raft_consensus.cc:2726] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260921 18:46:07.090112 7344 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
W20260921 18:46:07.090269 7214 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
I20260921 18:46:07.090335 7214 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: ae8442fea5d54cff8fccdb7b5d13fdb3; no voters: 5ef8886ab4d4417c8b0c251b1105d6ba, 7a966bfd1b9241fc807e119bbf645861, f367cedc80014733882b43e8a3c87d45
W20260921 18:46:07.090533 7475 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:07.090539 7972 raft_consensus.cc:2726] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260921 18:46:07.090595 7475 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba; no voters: ae8442fea5d54cff8fccdb7b5d13fdb3, af7b9688bd714366830a341b43b5bab9, f367cedc80014733882b43e8a3c87d45
W20260921 18:46:07.090672 7476 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
W20260921 18:46:07.090708 7475 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:07.090860 7973 raft_consensus.cc:2726] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260921 18:46:07.091082 7213 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:07.117972 3218 test_util.cc:268] Using random seed: -1447803252
I20260921 18:46:07.723378 3218 test_util.cc:268] Using random seed: -1447197846
I20260921 18:46:08.429363 3218 test_util.cc:268] Using random seed: -1446491869
I20260921 18:46:08.910583 7986 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:46:08.910755 7986 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:08.911340 7986 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:08.911722 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" is_pre_election: true
I20260921 18:46:08.911749 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45" is_pre_election: true
I20260921 18:46:08.911875 7278 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 3.
I20260921 18:46:08.911875 7409 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 3.
I20260921 18:46:08.912246 7474 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters:
I20260921 18:46:08.912423 7986 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260921 18:46:08.912503 7986 raft_consensus.cc:482] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:46:08.912592 7986 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 3 FOLLOWER]: Advancing to term 4
W20260921 18:46:08.912621 7475 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
W20260921 18:46:08.912902 7476 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
I20260921 18:46:08.913695 7986 raft_consensus.cc:504] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:08.914116 7986 leader_election.cc:288] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 election: Requested vote from peers f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667), ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775), 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431), af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:08.914314 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "f367cedc80014733882b43e8a3c87d45"
I20260921 18:46:08.914393 7278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d977daa5e3514a828d2098599ed2b135" candidate_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" candidate_term: 4 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3"
I20260921 18:46:08.914436 7409 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 3 FOLLOWER]: Advancing to term 4
I20260921 18:46:08.914500 7278 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 3 FOLLOWER]: Advancing to term 4
W20260921 18:46:08.915144 7476 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
I20260921 18:46:08.915503 7409 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 4.
I20260921 18:46:08.915503 7278 raft_consensus.cc:2454] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ef8886ab4d4417c8b0c251b1105d6ba in term 4.
W20260921 18:46:08.915611 7475 leader_election.cc:334] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111)
I20260921 18:46:08.915845 7476 leader_election.cc:302] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 5ef8886ab4d4417c8b0c251b1105d6ba, ae8442fea5d54cff8fccdb7b5d13fdb3, f367cedc80014733882b43e8a3c87d45; no voters: 7a966bfd1b9241fc807e119bbf645861, af7b9688bd714366830a341b43b5bab9
I20260921 18:46:08.915992 7986 raft_consensus.cc:2781] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 4 FOLLOWER]: Leader election won for term 4
I20260921 18:46:08.916196 7986 raft_consensus.cc:686] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [term 4 LEADER]: Becoming Leader. State: Replica: 5ef8886ab4d4417c8b0c251b1105d6ba, State: Running, Role: LEADER
I20260921 18:46:08.916522 7986 consensus_queue.cc:237] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:08.919631 7136 catalog_manager.cc:5495] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba reported cstate change: term changed from 2 to 4, leader changed from 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133) to 5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131). New cstate: current_term: 4 leader_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:09.235252 3218 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20260921 18:46:09.237740 7854 meta_cache.cc:302] tablet d977daa5e3514a828d2098599ed2b135: replica 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431) has failed: Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111)
W20260921 18:46:09.243870 7370 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32960: Illegal state: replica f367cedc80014733882b43e8a3c87d45 is not leader of this config: current role FOLLOWER
W20260921 18:46:09.249117 7239 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44584: Illegal state: replica ae8442fea5d54cff8fccdb7b5d13fdb3 is not leader of this config: current role FOLLOWER
I20260921 18:46:09.263562 7278 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P ae8442fea5d54cff8fccdb7b5d13fdb3 [term 4 FOLLOWER]: Refusing update from remote peer 5ef8886ab4d4417c8b0c251b1105d6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
I20260921 18:46:09.263698 7409 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P f367cedc80014733882b43e8a3c87d45 [term 4 FOLLOWER]: Refusing update from remote peer 5ef8886ab4d4417c8b0c251b1105d6ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 4 index: 14. (index mismatch)
W20260921 18:46:09.264036 7475 consensus_peers.cc:487] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba -> Peer af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): Couldn't send request to peer af7b9688bd714366830a341b43b5bab9. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.132:34763: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260921 18:46:09.264066 7990 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260921 18:46:09.264210 7476 consensus_peers.cc:487] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba -> Peer 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): Couldn't send request to peer 7a966bfd1b9241fc807e119bbf645861. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.133:35431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260921 18:46:09.264426 8001 consensus_queue.cc:1035] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260921 18:46:09.269273 8004 mvcc.cc:205] Tried to move back new op lower bound from 7331907048497856512 to 7331907047085187072. Current Snapshot: MvccSnapshot[applied={T|T < 7331907048497856512}]
I20260921 18:46:09.286911 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.133:35431
--local_ip_for_outbound_sockets=127.3.36.133
--tserver_master_addrs=127.3.36.190:42057
--webserver_port=36867
--webserver_interface=127.3.36.133
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260921 18:46:09.396234 8012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:09.396487 8012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:09.396553 8012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:09.400506 8012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:09.400645 8012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.133
I20260921 18:46:09.405730 8012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.133:35431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.3.36.133
--webserver_port=36867
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.8012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.133
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:09.406695 8012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:09.407994 8012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:09.416925 8018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:09.416968 8017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:09.417142 8012 server_base.cc:1034] running on GCE node
W20260921 18:46:09.416956 8020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:09.417737 8012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:09.418291 8012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:09.419467 8012 hybrid_clock.cc:648] HybridClock initialized: now 1790016369419452 us; error 26 us; skew 500 ppm
I20260921 18:46:09.425524 8012 webserver.cc:460] Webserver started at http://127.3.36.133:36867/ using document root <none> and password file <none>
I20260921 18:46:09.426070 8012 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:09.426186 8012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:09.429653 8012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260921 18:46:09.431622 8026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:09.432721 8012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:09.432848 8012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7a966bfd1b9241fc807e119bbf645861"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:09.433320 8012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:09.450393 8012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:09.450956 8012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:09.451140 8012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:09.451639 8012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:09.452665 8033 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260921 18:46:09.456266 8012 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260921 18:46:09.456336 8012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260921 18:46:09.456409 8012 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260921 18:46:09.457880 8012 ts_tablet_manager.cc:610] Registered 1 tablets
I20260921 18:46:09.457944 8012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:09.458181 8033 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Bootstrap starting.
I20260921 18:46:09.467780 8012 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.133:35431
I20260921 18:46:09.467813 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.133:35431 every 8 connection(s)
I20260921 18:46:09.468647 8012 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260921 18:46:09.472892 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8012
I20260921 18:46:09.473322 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.132:34763
--local_ip_for_outbound_sockets=127.3.36.132
--tserver_master_addrs=127.3.36.190:42057
--webserver_port=33247
--webserver_interface=127.3.36.132
--builtin_ntp_servers=127.3.36.148:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20260921 18:46:09.474530 8033 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:09.478858 8141 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:09.479104 8141 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:09.479542 8141 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:09.481238 7136 ts_manager.cc:194] Re-registered known tserver with Master: 7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431)
I20260921 18:46:09.482082 7136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.133:42221
I20260921 18:46:09.491895 8033 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260921 18:46:09.492296 8033 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Bootstrap complete.
I20260921 18:46:09.493673 8033 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Time spent bootstrapping tablet: real 0.036s user 0.024s sys 0.012s
I20260921 18:46:09.496609 8033 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:09.497131 8033 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a966bfd1b9241fc807e119bbf645861, State: Initialized, Role: FOLLOWER
I20260921 18:46:09.497635 8033 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:09.498663 8033 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:09.501794 8141 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
W20260921 18:46:09.590443 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:09.590695 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:09.590757 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:09.594709 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:09.594832 8145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.132
I20260921 18:46:09.599319 8145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.132:34763
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.36.132
--webserver_port=33247
--tserver_master_addrs=127.3.36.190:42057
--never_fsync=true
--heap_profile_path=/tmp/kudu.8145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.132
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:09.600253 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:09.601527 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:09.610734 8155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:09.610744 8153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:09.610797 8152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:09.611061 8145 server_base.cc:1034] running on GCE node
I20260921 18:46:09.611435 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:09.612061 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:09.613216 8145 hybrid_clock.cc:648] HybridClock initialized: now 1790016369613187 us; error 41 us; skew 500 ppm
I20260921 18:46:09.615062 8145 webserver.cc:460] Webserver started at http://127.3.36.132:33247/ using document root <none> and password file <none>
I20260921 18:46:09.615600 8145 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:09.615718 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:09.619158 8145 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:09.621106 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:09.622236 8145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:09.622355 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "af7b9688bd714366830a341b43b5bab9"
format_stamp: "Formatted at 2026-09-21 18:46:03 on dist-test-slave-l4st"
I20260921 18:46:09.622815 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:09.655799 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:09.656431 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:09.656603 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:09.657162 8145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:09.658290 8168 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260921 18:46:09.661911 8145 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260921 18:46:09.661988 8145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260921 18:46:09.662073 8145 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260921 18:46:09.663486 8145 ts_tablet_manager.cc:610] Registered 1 tablets
I20260921 18:46:09.663551 8145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:09.663865 8168 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Bootstrap starting.
I20260921 18:46:09.673695 8145 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.132:34763
I20260921 18:46:09.673756 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.132:34763 every 8 connection(s)
I20260921 18:46:09.674642 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260921 18:46:09.679272 8168 log.cc:824] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:09.680765 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8145
I20260921 18:46:09.684909 8276 heartbeater.cc:347] Connected to a master server at 127.3.36.190:42057
I20260921 18:46:09.685153 8276 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:09.685731 8276 heartbeater.cc:511] Master 127.3.36.190:42057 requested a full tablet report, sending...
I20260921 18:46:09.687534 7136 ts_manager.cc:194] Re-registered known tserver with Master: af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763)
I20260921 18:46:09.688285 7136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.132:35843
I20260921 18:46:09.690603 8168 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260921 18:46:09.691009 8168 tablet_bootstrap.cc:492] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Bootstrap complete.
I20260921 18:46:09.692097 8168 ts_tablet_manager.cc:1397] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20260921 18:46:09.694310 8168 raft_consensus.cc:348] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:09.694679 8168 raft_consensus.cc:729] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7b9688bd714366830a341b43b5bab9, State: Initialized, Role: FOLLOWER
I20260921 18:46:09.695091 8168 consensus_queue.cc:260] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f367cedc80014733882b43e8a3c87d45" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 42667 } } peers { permanent_uuid: "ae8442fea5d54cff8fccdb7b5d13fdb3" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 32775 } } peers { permanent_uuid: "7a966bfd1b9241fc807e119bbf645861" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 35431 } } peers { permanent_uuid: "5ef8886ab4d4417c8b0c251b1105d6ba" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 37089 } } peers { permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 } }
I20260921 18:46:09.695672 8276 heartbeater.cc:503] Master 127.3.36.190:42057 was elected leader, sending a full tablet report...
I20260921 18:46:09.695983 8168 ts_tablet_manager.cc:1428] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260921 18:46:09.709152 8230 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 1 FOLLOWER]: Advancing to term 4
I20260921 18:46:09.711326 8230 raft_consensus.cc:1261] T d977daa5e3514a828d2098599ed2b135 P af7b9688bd714366830a341b43b5bab9 [term 4 FOLLOWER]: Refusing update from remote peer 5ef8886ab4d4417c8b0c251b1105d6ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20260921 18:46:09.712363 7986 consensus_queue.cc:1037] T d977daa5e3514a828d2098599ed2b135 P 5ef8886ab4d4417c8b0c251b1105d6ba [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "af7b9688bd714366830a341b43b5bab9" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20260921 18:46:09.717440 8299 mvcc.cc:205] Tried to move back new op lower bound from 7331907048587051008 to 7331907033065295872. Current Snapshot: MvccSnapshot[applied={T|T < 7331907033300893696}]
I20260921 18:46:09.736452 7521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:46:09.743798 8210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.744288 7259 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.744305 8075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.749961 7390 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.764358 8095 raft_consensus.cc:3037] T d977daa5e3514a828d2098599ed2b135 P 7a966bfd1b9241fc807e119bbf645861 [term 2 FOLLOWER]: Advancing to term 4
I20260921 18:46:09.771360 8317 mvcc.cc:205] Tried to move back new op lower bound from 7331907048535957504 to 7331907047085187072. Current Snapshot: MvccSnapshot[applied={T|T < 7331907048497856512}]
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
249245e0706c462993d0cb93e01237a9 | 127.3.36.190:42057 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:42387 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5ef8886ab4d4417c8b0c251b1105d6ba | 127.3.36.131:37089 | HEALTHY | <none> | 1 | 0
7a966bfd1b9241fc807e119bbf645861 | 127.3.36.133:35431 | HEALTHY | <none> | 0 | 0
ae8442fea5d54cff8fccdb7b5d13fdb3 | 127.3.36.129:32775 | HEALTHY | <none> | 0 | 0
af7b9688bd714366830a341b43b5bab9 | 127.3.36.132:34763 | HEALTHY | <none> | 0 | 0
f367cedc80014733882b43e8a3c87d45 | 127.3.36.130:42667 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:32775
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:42667
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:37089
local_ip_for_outbound_sockets | 127.3.36.132 | experimental | 127.3.36.132:34763
local_ip_for_outbound_sockets | 127.3.36.133 | experimental | 127.3.36.133:35431
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:32775
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:42667
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:37089
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.36.132:34763
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.36.133:35431
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:42387 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet d977daa5e3514a828d2098599ed2b135 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
f367cedc80014733882b43e8a3c87d45 (127.3.36.130:42667): RUNNING
ae8442fea5d54cff8fccdb7b5d13fdb3 (127.3.36.129:32775): RUNNING
7a966bfd1b9241fc807e119bbf645861 (127.3.36.133:35431): RUNNING
5ef8886ab4d4417c8b0c251b1105d6ba (127.3.36.131:37089): RUNNING [LEADER]
af7b9688bd714366830a341b43b5bab9 (127.3.36.132:34763): RUNNING
All reported replicas are:
A = f367cedc80014733882b43e8a3c87d45
B = ae8442fea5d54cff8fccdb7b5d13fdb3
C = 7a966bfd1b9241fc807e119bbf645861
D = 5ef8886ab4d4417c8b0c251b1105d6ba
E = af7b9688bd714366830a341b43b5bab9
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D* E | | | Yes
A | A B C D* E | 4 | -1 | Yes
B | A B C D* E | 4 | -1 | Yes
C | A B C D E | 2 | -1 | Yes
D | A B C D* E | 4 | -1 | Yes
E | A B C D* E | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260921 18:46:09.782606 3218 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260921 18:46:09.983898 7521 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260921 18:46:09.984619 7259 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.984809 8210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.985437 8075 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260921 18:46:09.995909 7390 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
249245e0706c462993d0cb93e01237a9 | 127.3.36.190:42057 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:42387 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5ef8886ab4d4417c8b0c251b1105d6ba | 127.3.36.131:37089 | HEALTHY | <none> | 1 | 0
7a966bfd1b9241fc807e119bbf645861 | 127.3.36.133:35431 | HEALTHY | <none> | 0 | 0
ae8442fea5d54cff8fccdb7b5d13fdb3 | 127.3.36.129:32775 | HEALTHY | <none> | 0 | 0
af7b9688bd714366830a341b43b5bab9 | 127.3.36.132:34763 | HEALTHY | <none> | 0 | 0
f367cedc80014733882b43e8a3c87d45 | 127.3.36.130:42667 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.3.36.129 | experimental | 127.3.36.129:32775
local_ip_for_outbound_sockets | 127.3.36.130 | experimental | 127.3.36.130:42667
local_ip_for_outbound_sockets | 127.3.36.131 | experimental | 127.3.36.131:37089
local_ip_for_outbound_sockets | 127.3.36.132 | experimental | 127.3.36.132:34763
local_ip_for_outbound_sockets | 127.3.36.133 | experimental | 127.3.36.133:35431
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.36.129:32775
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.36.130:42667
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.36.131:37089
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.3.36.132:34763
server_dump_info_path | /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.3.36.133:35431
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.36.148:42387 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260921 18:46:10.018738 3218 log_verifier.cc:126] Checking tablet d977daa5e3514a828d2098599ed2b135
I20260921 18:46:10.060925 3218 log_verifier.cc:177] Verified matching terms for 18 ops in tablet d977daa5e3514a828d2098599ed2b135
I20260921 18:46:10.128973 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 7197
I20260921 18:46:10.138662 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 7328
I20260921 18:46:10.144042 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 7459
I20260921 18:46:10.150954 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8145
I20260921 18:46:10.156250 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8012
I20260921 18:46:10.161307 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 7106
2026-09-21T18:46:10Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (7634 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20260921 18:46:10.187554 3218 test_util.cc:268] Using random seed: -1444733666
I20260921 18:46:10.188350 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:10.188409 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:10.188431 3218 ts_itest-base.cc:114] 1 tablet servers
I20260921 18:46:10.188457 3218 ts_itest-base.cc:115] 1 replicas per TS
I20260921 18:46:10.188482 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:10Z Disabled control of system clock
I20260921 18:46:10.197041 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:46017
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:46017 with env {}
W20260921 18:46:10.302219 8365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:10.302603 8365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:10.302667 8365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:10.307461 8365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:10.307509 8365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:10.307528 8365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:10.307543 8365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:10.312179 8365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:46017
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:46017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:10.313099 8365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:10.314452 8365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:10.322038 8373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:10.322038 8370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:10.322057 8371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:10.322449 8365 server_base.cc:1034] running on GCE node
I20260921 18:46:10.323060 8365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:10.324054 8365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:10.325204 8365 hybrid_clock.cc:648] HybridClock initialized: now 1790016370325192 us; error 22 us; skew 500 ppm
I20260921 18:46:10.326983 8365 webserver.cc:460] Webserver started at http://127.3.36.190:34955/ using document root <none> and password file <none>
I20260921 18:46:10.327518 8365 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:10.327575 8365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:10.327817 8365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:10.329713 8365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "abf13c21f92a4abe8392d1bc6020f1aa"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.330237 8365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "abf13c21f92a4abe8392d1bc6020f1aa"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.333597 8365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260921 18:46:10.335670 8379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:10.336781 8365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:10.336910 8365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "abf13c21f92a4abe8392d1bc6020f1aa"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.337009 8365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:10.364811 8365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:10.365419 8365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:10.365625 8365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:10.371971 8365 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:46017
I20260921 18:46:10.371989 8431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:46017 every 8 connection(s)
I20260921 18:46:10.372818 8365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:10.375600 8432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:10.380306 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa: Bootstrap starting.
I20260921 18:46:10.382212 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8365
I20260921 18:46:10.382314 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:10.382771 8432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:10.383529 8432 log.cc:824] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:10.385245 8432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa: No bootstrap required, opened a new log
I20260921 18:46:10.388677 8432 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46017 } }
I20260921 18:46:10.388929 8432 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:10.388993 8432 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf13c21f92a4abe8392d1bc6020f1aa, State: Initialized, Role: FOLLOWER
I20260921 18:46:10.389446 8432 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46017 } }
I20260921 18:46:10.389673 8432 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:10.389747 8432 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:10.389838 8432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:10.390825 8432 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46017 } }
I20260921 18:46:10.391151 8432 leader_election.cc:302] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf13c21f92a4abe8392d1bc6020f1aa; no voters:
I20260921 18:46:10.391466 8432 leader_election.cc:288] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:10.391562 8437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:10.391758 8437 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [term 1 LEADER]: Becoming Leader. State: Replica: abf13c21f92a4abe8392d1bc6020f1aa, State: Running, Role: LEADER
I20260921 18:46:10.392028 8437 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46017 } }
I20260921 18:46:10.392639 8432 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:10.393545 8439 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf13c21f92a4abe8392d1bc6020f1aa. Latest consensus state: current_term: 1 leader_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46017 } } }
I20260921 18:46:10.393579 8438 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf13c21f92a4abe8392d1bc6020f1aa" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 46017 } } }
I20260921 18:46:10.393759 8438 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:10.393720 8439 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:10.394598 8446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:10.397454 8446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:10.402499 8446 catalog_manager.cc:1347] Generated new cluster ID: 075c3284212840c3a6c0f8c08615cca8
I20260921 18:46:10.402552 8446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:10.434355 8446 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:10.435238 8446 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:10.439116 8446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf13c21f92a4abe8392d1bc6020f1aa: Generated new TSK 0
I20260921 18:46:10.439736 8446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:10.444496 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:46017
--builtin_ntp_servers=127.3.36.148:43793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260921 18:46:10.548800 8456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:10.549064 8456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:10.549119 8456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:10.553084 8456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:10.553181 8456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:10.557483 8456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:43793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:46017
--never_fsync=true
--heap_profile_path=/tmp/kudu.8456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:10.558388 8456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:10.559685 8456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:10.568485 8464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:10.568464 8461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:10.568703 8456 server_base.cc:1034] running on GCE node
W20260921 18:46:10.568866 8462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:10.569231 8456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:10.569887 8456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:10.571074 8456 hybrid_clock.cc:648] HybridClock initialized: now 1790016370571058 us; error 28 us; skew 500 ppm
I20260921 18:46:10.572865 8456 webserver.cc:460] Webserver started at http://127.3.36.129:40907/ using document root <none> and password file <none>
I20260921 18:46:10.573414 8456 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:10.573524 8456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:10.573764 8456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:10.575603 8456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "69418220c5114161a81ca7edd731caa6"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.576134 8456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "69418220c5114161a81ca7edd731caa6"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.579531 8456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:10.581702 8470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:10.582823 8456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:10.582957 8456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "69418220c5114161a81ca7edd731caa6"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.583067 8456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:10.600176 8456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:10.600732 8456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:10.600907 8456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:10.601440 8456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:10.602356 8456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:10.602427 8456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:10.602494 8456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:10.602547 8456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:10.610849 8456 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:37907
I20260921 18:46:10.610873 8583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:37907 every 8 connection(s)
I20260921 18:46:10.611644 8456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:10.620092 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8456
I20260921 18:46:10.620219 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:10.621613 8584 heartbeater.cc:347] Connected to a master server at 127.3.36.190:46017
I20260921 18:46:10.621824 8584 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:10.622327 8584 heartbeater.cc:511] Master 127.3.36.190:46017 requested a full tablet report, sending...
I20260921 18:46:10.623612 8395 ts_manager.cc:194] Registered new tserver with Master: 69418220c5114161a81ca7edd731caa6 (127.3.36.129:37907)
I20260921 18:46:10.624622 8395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:55739
I20260921 18:46:10.634117 3218 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260921 18:46:10.642724 8395 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43908:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260921 18:46:10.658888 8518 tablet_service.cc:1467] Processing CreateTablet for tablet b67a8c87cffb41a08f20160874a6289f (DEFAULT_TABLE table=TestTable [id=81644766786b4e01b6e113040208ea2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:10.660002 8518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67a8c87cffb41a08f20160874a6289f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:10.665215 8594 tablet_bootstrap.cc:492] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6: Bootstrap starting.
I20260921 18:46:10.667023 8594 tablet_bootstrap.cc:654] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:10.667698 8594 log.cc:824] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:10.669371 8594 tablet_bootstrap.cc:492] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6: No bootstrap required, opened a new log
I20260921 18:46:10.669589 8594 ts_tablet_manager.cc:1397] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:10.672613 8594 raft_consensus.cc:348] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69418220c5114161a81ca7edd731caa6" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37907 } }
I20260921 18:46:10.672801 8594 raft_consensus.cc:374] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:10.672832 8594 raft_consensus.cc:729] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69418220c5114161a81ca7edd731caa6, State: Initialized, Role: FOLLOWER
I20260921 18:46:10.673153 8594 consensus_queue.cc:260] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69418220c5114161a81ca7edd731caa6" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37907 } }
I20260921 18:46:10.673257 8594 raft_consensus.cc:388] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:10.673301 8594 raft_consensus.cc:482] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:10.673365 8594 raft_consensus.cc:3037] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:10.674283 8594 raft_consensus.cc:504] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69418220c5114161a81ca7edd731caa6" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37907 } }
I20260921 18:46:10.674608 8594 leader_election.cc:302] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69418220c5114161a81ca7edd731caa6; no voters:
I20260921 18:46:10.674898 8594 leader_election.cc:288] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:10.675027 8596 raft_consensus.cc:2781] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:10.675266 8596 raft_consensus.cc:686] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [term 1 LEADER]: Becoming Leader. State: Replica: 69418220c5114161a81ca7edd731caa6, State: Running, Role: LEADER
I20260921 18:46:10.675477 8584 heartbeater.cc:503] Master 127.3.36.190:46017 was elected leader, sending a full tablet report...
I20260921 18:46:10.675593 8596 consensus_queue.cc:237] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69418220c5114161a81ca7edd731caa6" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37907 } }
I20260921 18:46:10.676230 8594 ts_tablet_manager.cc:1428] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260921 18:46:10.679034 8395 catalog_manager.cc:5495] T b67a8c87cffb41a08f20160874a6289f P 69418220c5114161a81ca7edd731caa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69418220c5114161a81ca7edd731caa6 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "69418220c5114161a81ca7edd731caa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69418220c5114161a81ca7edd731caa6" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 37907 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:10.691951 3218 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260921 18:46:10.693214 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 69418220c5114161a81ca7edd731caa6 to finish bootstrapping
I20260921 18:46:10.703111 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8456
I20260921 18:46:10.708732 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8365
2026-09-21T18:46:10Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (541 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20260921 18:46:10.728598 3218 test_util.cc:268] Using random seed: -1444192621
I20260921 18:46:10.729360 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:10.729425 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:10.729449 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:46:10.729463 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:46:10.729502 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:10Z Disabled control of system clock
I20260921 18:46:10.738111 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:37187
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:37187
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260921 18:46:10.847532 8609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:10.847816 8609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:10.847887 8609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:10.851845 8609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:10.851924 8609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:10.851958 8609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:10.852023 8609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:10.856595 8609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:37187
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:37187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:10.857676 8609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:10.859352 8609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:10.867300 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:10.867368 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:10.867522 8609 server_base.cc:1034] running on GCE node
W20260921 18:46:10.867370 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:10.868242 8609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:10.869282 8609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:10.870436 8609 hybrid_clock.cc:648] HybridClock initialized: now 1790016370870444 us; error 50 us; skew 500 ppm
I20260921 18:46:10.872267 8609 webserver.cc:460] Webserver started at http://127.3.36.190:38467/ using document root <none> and password file <none>
I20260921 18:46:10.872828 8609 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:10.872888 8609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:10.873113 8609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:10.874985 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "dde06ae7b9744bfeaf76dadb8dd63157"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.875516 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "dde06ae7b9744bfeaf76dadb8dd63157"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.878952 8609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260921 18:46:10.881114 8623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:10.882283 8609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:46:10.882416 8609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dde06ae7b9744bfeaf76dadb8dd63157"
format_stamp: "Formatted at 2026-09-21 18:46:10 on dist-test-slave-l4st"
I20260921 18:46:10.882519 8609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:10.895588 8609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:10.896150 8609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:10.896335 8609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:10.902684 8609 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:37187
I20260921 18:46:10.902706 8675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:37187 every 8 connection(s)
I20260921 18:46:10.903540 8609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:10.906527 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:10.911351 8676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157: Bootstrap starting.
I20260921 18:46:10.913401 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8609
I20260921 18:46:10.913522 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:10.913829 8676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:10.914716 8676 log.cc:824] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:10.916484 8676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157: No bootstrap required, opened a new log
I20260921 18:46:10.919729 8676 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37187 } }
I20260921 18:46:10.919950 8676 raft_consensus.cc:374] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:10.920003 8676 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde06ae7b9744bfeaf76dadb8dd63157, State: Initialized, Role: FOLLOWER
I20260921 18:46:10.920497 8676 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37187 } }
I20260921 18:46:10.920665 8676 raft_consensus.cc:388] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:10.920738 8676 raft_consensus.cc:482] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:10.920830 8676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:10.921662 8676 raft_consensus.cc:504] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37187 } }
I20260921 18:46:10.922039 8676 leader_election.cc:302] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dde06ae7b9744bfeaf76dadb8dd63157; no voters:
I20260921 18:46:10.922349 8676 leader_election.cc:288] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:10.922456 8681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:10.922701 8681 raft_consensus.cc:686] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [term 1 LEADER]: Becoming Leader. State: Replica: dde06ae7b9744bfeaf76dadb8dd63157, State: Running, Role: LEADER
I20260921 18:46:10.923012 8681 consensus_queue.cc:237] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37187 } }
I20260921 18:46:10.923470 8676 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:10.924379 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dde06ae7b9744bfeaf76dadb8dd63157. Latest consensus state: current_term: 1 leader_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37187 } } }
I20260921 18:46:10.924511 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:10.924897 8690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:10.925014 8683 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde06ae7b9744bfeaf76dadb8dd63157" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37187 } } }
I20260921 18:46:10.925150 8683 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:10.927910 8690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:10.933107 8690 catalog_manager.cc:1347] Generated new cluster ID: 36f44c834492411e95987aff4717a713
I20260921 18:46:10.933173 8690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:10.946705 8690 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:10.947630 8690 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:10.955133 8690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dde06ae7b9744bfeaf76dadb8dd63157: Generated new TSK 0
I20260921 18:46:10.955750 8690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:10.965461 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37187
--builtin_ntp_servers=127.3.36.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260921 18:46:11.073041 8700 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:11.073279 8700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:11.073298 8700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:11.073364 8700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:11.077320 8700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:11.077431 8700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:11.082449 8700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37187
--never_fsync=true
--heap_profile_path=/tmp/kudu.8700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:11.083407 8700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:11.084766 8700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:11.093906 8708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:11.093999 8700 server_base.cc:1034] running on GCE node
W20260921 18:46:11.093892 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:11.093892 8706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:11.094660 8700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:11.095263 8700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:11.096442 8700 hybrid_clock.cc:648] HybridClock initialized: now 1790016371096434 us; error 31 us; skew 500 ppm
I20260921 18:46:11.098222 8700 webserver.cc:460] Webserver started at http://127.3.36.129:39745/ using document root <none> and password file <none>
I20260921 18:46:11.098773 8700 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:11.098861 8700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:11.099089 8700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:11.100894 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b4ed22f1d9c94d7fbe90da6cf2947394"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.101435 8700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b4ed22f1d9c94d7fbe90da6cf2947394"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.104882 8700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:11.107142 8714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.108289 8700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:11.108413 8700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b4ed22f1d9c94d7fbe90da6cf2947394"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.108529 8700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:11.147990 8700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:11.148684 8700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:11.148877 8700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:11.149453 8700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:11.150417 8700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:11.150485 8700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.150568 8700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:11.150614 8700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.158849 8700 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:45847
I20260921 18:46:11.158880 8827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:45847 every 8 connection(s)
I20260921 18:46:11.159699 8700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:11.161629 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8700
I20260921 18:46:11.161752 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:11.164129 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37187
--builtin_ntp_servers=127.3.36.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:11.170855 8828 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37187
I20260921 18:46:11.171108 8828 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:11.171597 8828 heartbeater.cc:511] Master 127.3.36.190:37187 requested a full tablet report, sending...
I20260921 18:46:11.173020 8640 ts_manager.cc:194] Registered new tserver with Master: b4ed22f1d9c94d7fbe90da6cf2947394 (127.3.36.129:45847)
I20260921 18:46:11.174161 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:35815
W20260921 18:46:11.271883 8831 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:11.272162 8831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:11.272198 8831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:11.272289 8831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:11.276281 8831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:11.276420 8831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:11.280902 8831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37187
--never_fsync=true
--heap_profile_path=/tmp/kudu.8831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:11.281930 8831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:11.283200 8831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:11.292016 8836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:11.292073 8839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:11.292057 8837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:11.292302 8831 server_base.cc:1034] running on GCE node
I20260921 18:46:11.292642 8831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:11.293234 8831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:11.294381 8831 hybrid_clock.cc:648] HybridClock initialized: now 1790016371294354 us; error 39 us; skew 500 ppm
I20260921 18:46:11.296082 8831 webserver.cc:460] Webserver started at http://127.3.36.130:34719/ using document root <none> and password file <none>
I20260921 18:46:11.296617 8831 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:11.296679 8831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:11.296916 8831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:11.298770 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5636042622b143cf9dce9a5345715adc"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.299309 8831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5636042622b143cf9dce9a5345715adc"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.302635 8831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:11.304885 8845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.306197 8831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:11.306314 8831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5636042622b143cf9dce9a5345715adc"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.306434 8831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:11.328594 8831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:11.329207 8831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:11.329385 8831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:11.330034 8831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:11.330909 8831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:11.330953 8831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.331014 8831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:11.331048 8831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.339329 8831 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:39615
I20260921 18:46:11.339365 8958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:39615 every 8 connection(s)
I20260921 18:46:11.340129 8831 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:11.349221 8959 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37187
I20260921 18:46:11.349452 8959 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:11.349917 8959 heartbeater.cc:511] Master 127.3.36.190:37187 requested a full tablet report, sending...
I20260921 18:46:11.350294 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8831
I20260921 18:46:11.350411 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:11.350903 8640 ts_manager.cc:194] Registered new tserver with Master: 5636042622b143cf9dce9a5345715adc (127.3.36.130:39615)
I20260921 18:46:11.351711 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:38345
I20260921 18:46:11.352802 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37187
--builtin_ntp_servers=127.3.36.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260921 18:46:11.458491 8962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:11.458748 8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:11.458808 8962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:11.458873 8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:11.462818 8962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:11.462949 8962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:11.467628 8962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37187
--never_fsync=true
--heap_profile_path=/tmp/kudu.8962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:11.468575 8962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:11.469960 8962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:11.479770 8968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:11.479786 8970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:11.479790 8967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:11.479840 8962 server_base.cc:1034] running on GCE node
I20260921 18:46:11.480443 8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:11.480988 8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:11.482141 8962 hybrid_clock.cc:648] HybridClock initialized: now 1790016371482101 us; error 46 us; skew 500 ppm
I20260921 18:46:11.483906 8962 webserver.cc:460] Webserver started at http://127.3.36.131:46401/ using document root <none> and password file <none>
I20260921 18:46:11.484462 8962 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:11.484522 8962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:11.484773 8962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:11.486814 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9da2cca62c21480ba78eda527f650d97"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.487362 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9da2cca62c21480ba78eda527f650d97"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.491029 8962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:11.493419 8976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.494901 8962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:11.495026 8962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9da2cca62c21480ba78eda527f650d97"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.495162 8962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:11.505921 8962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:11.506513 8962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:11.506691 8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:11.507230 8962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:11.508111 8962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:11.508162 8962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.508224 8962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:11.508266 8962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.516796 8962 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:42681
I20260921 18:46:11.516868 9089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:42681 every 8 connection(s)
I20260921 18:46:11.517706 8962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:11.518226 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 8962
I20260921 18:46:11.518330 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:11.527601 9090 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37187
I20260921 18:46:11.527832 9090 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:11.528355 9090 heartbeater.cc:511] Master 127.3.36.190:37187 requested a full tablet report, sending...
I20260921 18:46:11.529302 8639 ts_manager.cc:194] Registered new tserver with Master: 9da2cca62c21480ba78eda527f650d97 (127.3.36.131:42681)
I20260921 18:46:11.529891 8639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:39743
I20260921 18:46:11.532389 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:11.540997 8639 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39888:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260921 18:46:11.543857 8639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:46:11.562525 8893 tablet_service.cc:1467] Processing CreateTablet for tablet 9368dd0fd6fa472a873ae45a0a12ebe1 (DEFAULT_TABLE table=TestTable [id=39403ba4c535447da8861ce158523e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:11.563212 9024 tablet_service.cc:1467] Processing CreateTablet for tablet 9368dd0fd6fa472a873ae45a0a12ebe1 (DEFAULT_TABLE table=TestTable [id=39403ba4c535447da8861ce158523e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:11.563612 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9368dd0fd6fa472a873ae45a0a12ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:11.563349 8762 tablet_service.cc:1467] Processing CreateTablet for tablet 9368dd0fd6fa472a873ae45a0a12ebe1 (DEFAULT_TABLE table=TestTable [id=39403ba4c535447da8861ce158523e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:11.564163 9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9368dd0fd6fa472a873ae45a0a12ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:11.564178 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9368dd0fd6fa472a873ae45a0a12ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:11.569363 9107 tablet_bootstrap.cc:492] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394: Bootstrap starting.
I20260921 18:46:11.570504 9108 tablet_bootstrap.cc:492] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97: Bootstrap starting.
I20260921 18:46:11.571864 9107 tablet_bootstrap.cc:654] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:11.572535 9108 tablet_bootstrap.cc:654] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:11.572907 9107 log.cc:824] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:11.573313 9108 log.cc:824] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:11.573599 9109 tablet_bootstrap.cc:492] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: Bootstrap starting.
I20260921 18:46:11.574676 9107 tablet_bootstrap.cc:492] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394: No bootstrap required, opened a new log
I20260921 18:46:11.574870 9107 ts_tablet_manager.cc:1397] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260921 18:46:11.575068 9108 tablet_bootstrap.cc:492] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97: No bootstrap required, opened a new log
I20260921 18:46:11.575251 9108 ts_tablet_manager.cc:1397] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:46:11.575556 9109 tablet_bootstrap.cc:654] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:11.576280 9109 log.cc:824] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:11.578068 9109 tablet_bootstrap.cc:492] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: No bootstrap required, opened a new log
I20260921 18:46:11.578240 9109 ts_tablet_manager.cc:1397] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:11.578513 9107 raft_consensus.cc:348] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.578728 9107 raft_consensus.cc:729] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ed22f1d9c94d7fbe90da6cf2947394, State: Initialized, Role: FOLLOWER
I20260921 18:46:11.579030 9107 consensus_queue.cc:260] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.579536 9108 raft_consensus.cc:348] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.579802 9108 raft_consensus.cc:729] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da2cca62c21480ba78eda527f650d97, State: Initialized, Role: FOLLOWER
I20260921 18:46:11.579993 9107 ts_tablet_manager.cc:1428] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:46:11.580134 9108 consensus_queue.cc:260] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.580328 8828 heartbeater.cc:503] Master 127.3.36.190:37187 was elected leader, sending a full tablet report...
I20260921 18:46:11.580722 9090 heartbeater.cc:503] Master 127.3.36.190:37187 was elected leader, sending a full tablet report...
I20260921 18:46:11.581001 9108 ts_tablet_manager.cc:1428] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:46:11.582404 9109 raft_consensus.cc:348] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.582638 9109 raft_consensus.cc:729] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5636042622b143cf9dce9a5345715adc, State: Initialized, Role: FOLLOWER
I20260921 18:46:11.582885 9109 consensus_queue.cc:260] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.583439 8959 heartbeater.cc:503] Master 127.3.36.190:37187 was elected leader, sending a full tablet report...
I20260921 18:46:11.583741 9109 ts_tablet_manager.cc:1428] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:46:11.590571 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260921 18:46:11.590926 8960 tablet.cc:2369] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:11.591904 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b4ed22f1d9c94d7fbe90da6cf2947394 to finish bootstrapping
I20260921 18:46:11.597234 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5636042622b143cf9dce9a5345715adc to finish bootstrapping
I20260921 18:46:11.602313 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9da2cca62c21480ba78eda527f650d97 to finish bootstrapping
I20260921 18:46:11.612106 8913 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9368dd0fd6fa472a873ae45a0a12ebe1"
dest_uuid: "5636042622b143cf9dce9a5345715adc"
from {username='slave'} at 127.0.0.1:41124
I20260921 18:46:11.612326 8913 raft_consensus.cc:482] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:11.612452 8913 raft_consensus.cc:3037] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:11.613718 8913 raft_consensus.cc:504] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.614646 8913 leader_election.cc:288] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [CANDIDATE]: Term 1 election: Requested vote from peers 9da2cca62c21480ba78eda527f650d97 (127.3.36.131:42681), b4ed22f1d9c94d7fbe90da6cf2947394 (127.3.36.129:45847)
I20260921 18:46:11.618782 8782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9368dd0fd6fa472a873ae45a0a12ebe1" candidate_uuid: "5636042622b143cf9dce9a5345715adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394"
I20260921 18:46:11.619027 8782 raft_consensus.cc:3037] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:11.619378 9044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9368dd0fd6fa472a873ae45a0a12ebe1" candidate_uuid: "5636042622b143cf9dce9a5345715adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9da2cca62c21480ba78eda527f650d97"
I20260921 18:46:11.619570 9044 raft_consensus.cc:3037] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:11.620266 8782 raft_consensus.cc:2454] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5636042622b143cf9dce9a5345715adc in term 1.
I20260921 18:46:11.620703 8847 leader_election.cc:302] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5636042622b143cf9dce9a5345715adc, b4ed22f1d9c94d7fbe90da6cf2947394; no voters:
I20260921 18:46:11.620822 9044 raft_consensus.cc:2454] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5636042622b143cf9dce9a5345715adc in term 1.
I20260921 18:46:11.620980 9115 raft_consensus.cc:2781] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:11.621207 9115 raft_consensus.cc:686] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 LEADER]: Becoming Leader. State: Replica: 5636042622b143cf9dce9a5345715adc, State: Running, Role: LEADER
I20260921 18:46:11.621562 9115 consensus_queue.cc:237] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.624683 8638 catalog_manager.cc:5495] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5636042622b143cf9dce9a5345715adc (127.3.36.130). New cstate: current_term: 1 leader_uuid: "5636042622b143cf9dce9a5345715adc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:11.630730 9044 raft_consensus.cc:1261] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 9da2cca62c21480ba78eda527f650d97 [term 1 FOLLOWER]: Refusing update from remote peer 5636042622b143cf9dce9a5345715adc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:11.630730 8782 raft_consensus.cc:1261] T 9368dd0fd6fa472a873ae45a0a12ebe1 P b4ed22f1d9c94d7fbe90da6cf2947394 [term 1 FOLLOWER]: Refusing update from remote peer 5636042622b143cf9dce9a5345715adc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:11.631314 9119 consensus_queue.cc:1035] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:11.631459 9115 consensus_queue.cc:1035] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:11.639708 9123 mvcc.cc:205] Tried to move back new op lower bound from 7331907058193326080 to 7331907058164633600. Current Snapshot: MvccSnapshot[applied={T|T < 7331907058193326080}]
I20260921 18:46:11.650439 8913 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9368dd0fd6fa472a873ae45a0a12ebe1"
dest_uuid: "5636042622b143cf9dce9a5345715adc"
from {username='slave'} at 127.0.0.1:41124
I20260921 18:46:11.650607 8913 raft_consensus.cc:580] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 LEADER]: Received request to step down
I20260921 18:46:11.650681 8913 raft_consensus.cc:3032] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:11.650722 8913 raft_consensus.cc:729] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5636042622b143cf9dce9a5345715adc, State: Running, Role: LEADER
I20260921 18:46:11.650851 8913 consensus_queue.cc:260] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5636042622b143cf9dce9a5345715adc" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 39615 } } peers { permanent_uuid: "9da2cca62c21480ba78eda527f650d97" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42681 } } peers { permanent_uuid: "b4ed22f1d9c94d7fbe90da6cf2947394" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 45847 } }
I20260921 18:46:11.651131 8913 raft_consensus.cc:3037] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:11.652361 8913 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9368dd0fd6fa472a873ae45a0a12ebe1"
dest_uuid: "5636042622b143cf9dce9a5345715adc"
from {username='slave'} at 127.0.0.1:41124
I20260921 18:46:11.652479 8913 raft_consensus.cc:573] T 9368dd0fd6fa472a873ae45a0a12ebe1 P 5636042622b143cf9dce9a5345715adc [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20260921 18:46:11.653693 8873 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41124: Illegal state: replica 5636042622b143cf9dce9a5345715adc is not leader of this config: current role FOLLOWER
I20260921 18:46:11.654417 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8700
I20260921 18:46:11.661262 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8831
I20260921 18:46:11.666970 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8962
I20260921 18:46:11.672264 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 8609
2026-09-21T18:46:11Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (964 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20260921 18:46:11.693894 3218 test_util.cc:268] Using random seed: -1443227327
I20260921 18:46:11.694787 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:11.694826 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:11.694844 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:46:11.694859 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:46:11.694872 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:11Z Disabled control of system clock
I20260921 18:46:11.708959 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:35621
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:35621
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260921 18:46:11.836289 9136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:11.836531 9136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:11.836563 9136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:11.840448 9136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:11.840492 9136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:11.840509 9136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:11.840525 9136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:11.845000 9136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:35621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:11.845934 9136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:11.847172 9136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:11.855096 9141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:11.855096 9142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:11.855297 9136 server_base.cc:1034] running on GCE node
W20260921 18:46:11.855146 9144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:11.855855 9136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:11.856797 9136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:11.857950 9136 hybrid_clock.cc:648] HybridClock initialized: now 1790016371857932 us; error 30 us; skew 500 ppm
I20260921 18:46:11.859758 9136 webserver.cc:460] Webserver started at http://127.3.36.190:45345/ using document root <none> and password file <none>
I20260921 18:46:11.860292 9136 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:11.860350 9136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:11.860585 9136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:11.862438 9136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9f925eb40e9248fc87ffc8ab94631217"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.863010 9136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9f925eb40e9248fc87ffc8ab94631217"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.866542 9136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:11.868904 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:11.870076 9136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:11.870223 9136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9f925eb40e9248fc87ffc8ab94631217"
format_stamp: "Formatted at 2026-09-21 18:46:11 on dist-test-slave-l4st"
I20260921 18:46:11.870323 9136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:11.894547 9136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:11.895217 9136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:11.895403 9136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:11.901889 9136 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:35621
I20260921 18:46:11.901897 9202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:35621 every 8 connection(s)
I20260921 18:46:11.902791 9136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:11.905200 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9136
I20260921 18:46:11.905295 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:11.906128 9203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:11.912560 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217: Bootstrap starting.
I20260921 18:46:11.915148 9203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:11.915957 9203 log.cc:824] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:11.917809 9203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217: No bootstrap required, opened a new log
I20260921 18:46:11.921403 9203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f925eb40e9248fc87ffc8ab94631217" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 35621 } }
I20260921 18:46:11.921665 9203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:11.921712 9203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f925eb40e9248fc87ffc8ab94631217, State: Initialized, Role: FOLLOWER
I20260921 18:46:11.922097 9203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f925eb40e9248fc87ffc8ab94631217" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 35621 } }
I20260921 18:46:11.922210 9203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:11.922254 9203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:11.922322 9203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:11.923014 9203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f925eb40e9248fc87ffc8ab94631217" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 35621 } }
I20260921 18:46:11.923372 9203 leader_election.cc:302] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f925eb40e9248fc87ffc8ab94631217; no voters:
I20260921 18:46:11.923728 9203 leader_election.cc:288] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:11.923770 9208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:11.924038 9208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [term 1 LEADER]: Becoming Leader. State: Replica: 9f925eb40e9248fc87ffc8ab94631217, State: Running, Role: LEADER
I20260921 18:46:11.924345 9208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f925eb40e9248fc87ffc8ab94631217" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 35621 } }
I20260921 18:46:11.924872 9203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:11.926098 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f925eb40e9248fc87ffc8ab94631217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f925eb40e9248fc87ffc8ab94631217" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 35621 } } }
I20260921 18:46:11.926235 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:11.926923 9217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:11.927126 9210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f925eb40e9248fc87ffc8ab94631217. Latest consensus state: current_term: 1 leader_uuid: "9f925eb40e9248fc87ffc8ab94631217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f925eb40e9248fc87ffc8ab94631217" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 35621 } } }
I20260921 18:46:11.927215 9210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:11.929682 9217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:11.934528 9217 catalog_manager.cc:1347] Generated new cluster ID: 6082e01366614a84b55f330348c2b00d
I20260921 18:46:11.934607 9217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:11.945827 9217 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:11.946892 9217 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:11.954931 9217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f925eb40e9248fc87ffc8ab94631217: Generated new TSK 0
I20260921 18:46:11.955528 9217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:11.958000 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:35621
--builtin_ntp_servers=127.3.36.148:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20260921 18:46:12.062471 9227 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:12.062705 9227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:12.062726 9227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:12.062798 9227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:12.066767 9227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:12.066845 9227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:12.071472 9227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:35621
--never_fsync=true
--heap_profile_path=/tmp/kudu.9227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:12.072409 9227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:12.073820 9227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:12.082839 9232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:12.082826 9235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:12.082823 9233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:12.083181 9227 server_base.cc:1034] running on GCE node
I20260921 18:46:12.083578 9227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:12.084124 9227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:12.085321 9227 hybrid_clock.cc:648] HybridClock initialized: now 1790016372085307 us; error 25 us; skew 500 ppm
I20260921 18:46:12.087110 9227 webserver.cc:460] Webserver started at http://127.3.36.129:33795/ using document root <none> and password file <none>
I20260921 18:46:12.087663 9227 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:12.087749 9227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:12.087980 9227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:12.089830 9227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "370e810d2c1f42238704e676728bbf03"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.090372 9227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "370e810d2c1f42238704e676728bbf03"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.093838 9227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:12.096113 9241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.097198 9227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:12.097321 9227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "370e810d2c1f42238704e676728bbf03"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.097420 9227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:12.122191 9227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:12.122821 9227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:12.123020 9227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:12.123647 9227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:12.124569 9227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:12.124634 9227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.124691 9227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:12.124742 9227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.132903 9227 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:39111
I20260921 18:46:12.132974 9354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:39111 every 8 connection(s)
I20260921 18:46:12.133823 9227 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:12.143151 9355 heartbeater.cc:347] Connected to a master server at 127.3.36.190:35621
I20260921 18:46:12.143380 9355 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:12.143774 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9227
I20260921 18:46:12.143839 9355 heartbeater.cc:511] Master 127.3.36.190:35621 requested a full tablet report, sending...
I20260921 18:46:12.143922 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:12.145345 9167 ts_manager.cc:194] Registered new tserver with Master: 370e810d2c1f42238704e676728bbf03 (127.3.36.129:39111)
I20260921 18:46:12.146263 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:35621
--builtin_ntp_servers=127.3.36.148:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260921 18:46:12.146943 9167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:41725
W20260921 18:46:12.253391 9358 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:12.253721 9358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:12.253768 9358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:12.253829 9358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:12.257957 9358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:12.258097 9358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:12.262817 9358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:35621
--never_fsync=true
--heap_profile_path=/tmp/kudu.9358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:12.263852 9358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:12.265251 9358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:12.274180 9366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:12.274185 9363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:12.274226 9364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:12.274507 9358 server_base.cc:1034] running on GCE node
I20260921 18:46:12.274921 9358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:12.275480 9358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:12.276656 9358 hybrid_clock.cc:648] HybridClock initialized: now 1790016372276631 us; error 44 us; skew 500 ppm
I20260921 18:46:12.278355 9358 webserver.cc:460] Webserver started at http://127.3.36.130:44929/ using document root <none> and password file <none>
I20260921 18:46:12.278906 9358 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:12.279006 9358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:12.279242 9358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:12.281026 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1234ad2f0c704872adc75af9928c168e"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.281579 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1234ad2f0c704872adc75af9928c168e"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.285043 9358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:12.287266 9372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.288403 9358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:12.288489 9358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1234ad2f0c704872adc75af9928c168e"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.288585 9358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:12.315258 9358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:12.315899 9358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:12.316043 9358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:12.316571 9358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:12.317431 9358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:12.317525 9358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.317600 9358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:12.317624 9358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.326723 9358 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:33475
I20260921 18:46:12.326777 9485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:33475 every 8 connection(s)
I20260921 18:46:12.327620 9358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:12.332625 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9358
I20260921 18:46:12.332718 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:12.334781 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:35621
--builtin_ntp_servers=127.3.36.148:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20260921 18:46:12.338027 9486 heartbeater.cc:347] Connected to a master server at 127.3.36.190:35621
I20260921 18:46:12.338258 9486 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:12.338800 9486 heartbeater.cc:511] Master 127.3.36.190:35621 requested a full tablet report, sending...
I20260921 18:46:12.339847 9167 ts_manager.cc:194] Registered new tserver with Master: 1234ad2f0c704872adc75af9928c168e (127.3.36.130:33475)
I20260921 18:46:12.340696 9167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:39969
W20260921 18:46:12.442281 9489 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:12.442515 9489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:12.442534 9489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:12.442566 9489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:12.446432 9489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:12.446509 9489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:12.450960 9489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:35621
--never_fsync=true
--heap_profile_path=/tmp/kudu.9489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:12.451812 9489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:12.453130 9489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:12.462239 9497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:12.462268 9489 server_base.cc:1034] running on GCE node
W20260921 18:46:12.462210 9495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:12.462184 9494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:12.462960 9489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:12.463524 9489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:12.464704 9489 hybrid_clock.cc:648] HybridClock initialized: now 1790016372464689 us; error 24 us; skew 500 ppm
I20260921 18:46:12.466377 9489 webserver.cc:460] Webserver started at http://127.3.36.131:38563/ using document root <none> and password file <none>
I20260921 18:46:12.466905 9489 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:12.466988 9489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:12.467197 9489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:12.468914 9489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "44a3cc8eaed441db87f3b70515583a4b"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.469425 9489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "44a3cc8eaed441db87f3b70515583a4b"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.472720 9489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260921 18:46:12.474937 9503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.476045 9489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:12.476177 9489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "44a3cc8eaed441db87f3b70515583a4b"
format_stamp: "Formatted at 2026-09-21 18:46:12 on dist-test-slave-l4st"
I20260921 18:46:12.476279 9489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:12.493434 9489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:12.494094 9489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:12.494269 9489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:12.494824 9489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:12.495627 9489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:12.495674 9489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.495730 9489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:12.495762 9489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:12.503947 9489 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:42163
I20260921 18:46:12.503969 9616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:42163 every 8 connection(s)
I20260921 18:46:12.504783 9489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:12.510902 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9489
I20260921 18:46:12.511018 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:12.514644 9617 heartbeater.cc:347] Connected to a master server at 127.3.36.190:35621
I20260921 18:46:12.514818 9617 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:12.515220 9617 heartbeater.cc:511] Master 127.3.36.190:35621 requested a full tablet report, sending...
I20260921 18:46:12.516038 9167 ts_manager.cc:194] Registered new tserver with Master: 44a3cc8eaed441db87f3b70515583a4b (127.3.36.131:42163)
I20260921 18:46:12.516569 9167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:54443
I20260921 18:46:12.524729 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:12.533448 9167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42292:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260921 18:46:12.536316 9167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:46:12.552801 9551 tablet_service.cc:1467] Processing CreateTablet for tablet 393e167d65294f738b5cfb907b1d0d43 (DEFAULT_TABLE table=TestTable [id=8f0585011ac942a982e65e2367751072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:12.553143 9420 tablet_service.cc:1467] Processing CreateTablet for tablet 393e167d65294f738b5cfb907b1d0d43 (DEFAULT_TABLE table=TestTable [id=8f0585011ac942a982e65e2367751072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:12.553980 9551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393e167d65294f738b5cfb907b1d0d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:12.553635 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 393e167d65294f738b5cfb907b1d0d43 (DEFAULT_TABLE table=TestTable [id=8f0585011ac942a982e65e2367751072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:12.554114 9420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393e167d65294f738b5cfb907b1d0d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:12.554543 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393e167d65294f738b5cfb907b1d0d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:12.559620 9634 tablet_bootstrap.cc:492] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: Bootstrap starting.
I20260921 18:46:12.561028 9636 tablet_bootstrap.cc:492] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: Bootstrap starting.
I20260921 18:46:12.561625 9635 tablet_bootstrap.cc:492] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b: Bootstrap starting.
I20260921 18:46:12.561826 9634 tablet_bootstrap.cc:654] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:12.562484 9634 log.cc:824] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:12.563126 9636 tablet_bootstrap.cc:654] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:12.563593 9635 tablet_bootstrap.cc:654] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:12.563901 9636 log.cc:824] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:12.564175 9634 tablet_bootstrap.cc:492] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: No bootstrap required, opened a new log
I20260921 18:46:12.564394 9634 ts_tablet_manager.cc:1397] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:46:12.564378 9635 log.cc:824] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:12.565750 9636 tablet_bootstrap.cc:492] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: No bootstrap required, opened a new log
I20260921 18:46:12.565951 9636 ts_tablet_manager.cc:1397] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:12.566159 9635 tablet_bootstrap.cc:492] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b: No bootstrap required, opened a new log
I20260921 18:46:12.566319 9635 ts_tablet_manager.cc:1397] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:12.568813 9634 raft_consensus.cc:348] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.569087 9634 raft_consensus.cc:729] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1234ad2f0c704872adc75af9928c168e, State: Initialized, Role: FOLLOWER
I20260921 18:46:12.569018 9636 raft_consensus.cc:348] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.569308 9636 raft_consensus.cc:729] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370e810d2c1f42238704e676728bbf03, State: Initialized, Role: FOLLOWER
I20260921 18:46:12.569403 9634 consensus_queue.cc:260] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.569646 9636 consensus_queue.cc:260] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.570111 9486 heartbeater.cc:503] Master 127.3.36.190:35621 was elected leader, sending a full tablet report...
I20260921 18:46:12.570364 9634 ts_tablet_manager.cc:1428] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260921 18:46:12.570364 9635 raft_consensus.cc:348] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.570587 9635 raft_consensus.cc:729] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a3cc8eaed441db87f3b70515583a4b, State: Initialized, Role: FOLLOWER
I20260921 18:46:12.570868 9635 consensus_queue.cc:260] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.571197 9636 ts_tablet_manager.cc:1428] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:12.571516 9355 heartbeater.cc:503] Master 127.3.36.190:35621 was elected leader, sending a full tablet report...
I20260921 18:46:12.571668 9635 ts_tablet_manager.cc:1428] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:46:12.571925 9617 heartbeater.cc:503] Master 127.3.36.190:35621 was elected leader, sending a full tablet report...
W20260921 18:46:12.578554 9487 tablet.cc:2369] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:12.590440 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:12.591814 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 370e810d2c1f42238704e676728bbf03 to finish bootstrapping
I20260921 18:46:12.598793 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1234ad2f0c704872adc75af9928c168e to finish bootstrapping
I20260921 18:46:12.604323 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 44a3cc8eaed441db87f3b70515583a4b to finish bootstrapping
I20260921 18:46:12.614931 9440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "393e167d65294f738b5cfb907b1d0d43"
dest_uuid: "1234ad2f0c704872adc75af9928c168e"
from {username='slave'} at 127.0.0.1:47572
I20260921 18:46:12.615136 9440 raft_consensus.cc:482] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:12.615288 9440 raft_consensus.cc:3037] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:12.616253 9440 raft_consensus.cc:504] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.617239 9440 leader_election.cc:288] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [CANDIDATE]: Term 1 election: Requested vote from peers 44a3cc8eaed441db87f3b70515583a4b (127.3.36.131:42163), 370e810d2c1f42238704e676728bbf03 (127.3.36.129:39111)
I20260921 18:46:12.621743 9571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393e167d65294f738b5cfb907b1d0d43" candidate_uuid: "1234ad2f0c704872adc75af9928c168e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44a3cc8eaed441db87f3b70515583a4b"
I20260921 18:46:12.621961 9571 raft_consensus.cc:3037] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:12.623021 9571 raft_consensus.cc:2454] T 393e167d65294f738b5cfb907b1d0d43 P 44a3cc8eaed441db87f3b70515583a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1234ad2f0c704872adc75af9928c168e in term 1.
I20260921 18:46:12.623428 9376 leader_election.cc:302] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1234ad2f0c704872adc75af9928c168e, 44a3cc8eaed441db87f3b70515583a4b; no voters:
I20260921 18:46:12.623632 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393e167d65294f738b5cfb907b1d0d43" candidate_uuid: "1234ad2f0c704872adc75af9928c168e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "370e810d2c1f42238704e676728bbf03"
I20260921 18:46:12.623806 9640 raft_consensus.cc:2781] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:12.623844 9309 raft_consensus.cc:3037] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:12.624019 9640 raft_consensus.cc:686] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 LEADER]: Becoming Leader. State: Replica: 1234ad2f0c704872adc75af9928c168e, State: Running, Role: LEADER
I20260921 18:46:12.624330 9640 consensus_queue.cc:237] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:12.625000 9309 raft_consensus.cc:2454] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1234ad2f0c704872adc75af9928c168e in term 1.
I20260921 18:46:12.627506 9165 catalog_manager.cc:5495] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1234ad2f0c704872adc75af9928c168e (127.3.36.130). New cstate: current_term: 1 leader_uuid: "1234ad2f0c704872adc75af9928c168e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } health_report { overall_health: HEALTHY } } }
W20260921 18:46:12.634912 9356 tablet.cc:2369] T 393e167d65294f738b5cfb907b1d0d43 P 370e810d2c1f42238704e676728bbf03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:13.653934 9440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "393e167d65294f738b5cfb907b1d0d43"
dest_uuid: "1234ad2f0c704872adc75af9928c168e"
from {username='slave'} at 127.0.0.1:47572
I20260921 18:46:13.654170 9440 raft_consensus.cc:580] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 LEADER]: Received request to step down
I20260921 18:46:13.654240 9440 raft_consensus.cc:3032] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:13.654304 9440 raft_consensus.cc:729] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1234ad2f0c704872adc75af9928c168e, State: Running, Role: LEADER
I20260921 18:46:13.654456 9440 consensus_queue.cc:260] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a3cc8eaed441db87f3b70515583a4b" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 42163 } } peers { permanent_uuid: "370e810d2c1f42238704e676728bbf03" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 39111 } } peers { permanent_uuid: "1234ad2f0c704872adc75af9928c168e" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 33475 } }
I20260921 18:46:13.654597 9440 raft_consensus.cc:3037] T 393e167d65294f738b5cfb907b1d0d43 P 1234ad2f0c704872adc75af9928c168e [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:13.655380 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9227
W20260921 18:46:13.660389 9374 connection.cc:535] client connection to 127.3.36.129:39111 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260921 18:46:13.660578 9374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:46:13.660710 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9358
I20260921 18:46:13.665884 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9489
I20260921 18:46:13.671046 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9136
2026-09-21T18:46:13Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (1997 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20260921 18:46:13.691295 3218 test_util.cc:268] Using random seed: -1441229924
I20260921 18:46:13.692050 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:13.692113 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:13.692133 3218 ts_itest-base.cc:114] 5 tablet servers
I20260921 18:46:13.692160 3218 ts_itest-base.cc:115] 5 replicas per TS
I20260921 18:46:13.692184 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:13Z Disabled control of system clock
I20260921 18:46:13.700374 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36131
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:36131
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260921 18:46:13.805199 9655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:13.805478 9655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:13.805576 9655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:13.809762 9655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:13.809831 9655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:13.809864 9655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:13.809921 9655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:13.814602 9655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:36131
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:13.815591 9655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:13.816921 9655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:13.824764 9660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:13.824689 9661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:13.824725 9663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:13.825002 9655 server_base.cc:1034] running on GCE node
I20260921 18:46:13.825664 9655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:13.826669 9655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:13.827857 9655 hybrid_clock.cc:648] HybridClock initialized: now 1790016373827830 us; error 37 us; skew 500 ppm
I20260921 18:46:13.829582 9655 webserver.cc:460] Webserver started at http://127.3.36.190:44773/ using document root <none> and password file <none>
I20260921 18:46:13.830111 9655 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:13.830197 9655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:13.830399 9655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:13.832146 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "852aedb0dca9415da7df9feecc7170af"
format_stamp: "Formatted at 2026-09-21 18:46:13 on dist-test-slave-l4st"
I20260921 18:46:13.832664 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "852aedb0dca9415da7df9feecc7170af"
format_stamp: "Formatted at 2026-09-21 18:46:13 on dist-test-slave-l4st"
I20260921 18:46:13.835915 9655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260921 18:46:13.838064 9669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:13.839148 9655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:13.839280 9655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "852aedb0dca9415da7df9feecc7170af"
format_stamp: "Formatted at 2026-09-21 18:46:13 on dist-test-slave-l4st"
I20260921 18:46:13.839378 9655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:13.862355 9655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:13.862944 9655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:13.863138 9655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:13.869370 9655 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:36131
I20260921 18:46:13.869383 9721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:36131 every 8 connection(s)
I20260921 18:46:13.870270 9655 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:13.873133 9722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:13.875648 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9655
I20260921 18:46:13.875758 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:13.878880 9722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af: Bootstrap starting.
I20260921 18:46:13.880852 9722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:13.881592 9722 log.cc:824] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:13.883417 9722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af: No bootstrap required, opened a new log
I20260921 18:46:13.886942 9722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852aedb0dca9415da7df9feecc7170af" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36131 } }
I20260921 18:46:13.887131 9722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:13.887198 9722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852aedb0dca9415da7df9feecc7170af, State: Initialized, Role: FOLLOWER
I20260921 18:46:13.887627 9722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852aedb0dca9415da7df9feecc7170af" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36131 } }
I20260921 18:46:13.887768 9722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:13.887850 9722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:13.887975 9722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:13.888695 9722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852aedb0dca9415da7df9feecc7170af" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36131 } }
I20260921 18:46:13.889004 9722 leader_election.cc:302] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 852aedb0dca9415da7df9feecc7170af; no voters:
I20260921 18:46:13.889287 9722 leader_election.cc:288] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:13.889410 9727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:13.889676 9727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [term 1 LEADER]: Becoming Leader. State: Replica: 852aedb0dca9415da7df9feecc7170af, State: Running, Role: LEADER
I20260921 18:46:13.890203 9727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852aedb0dca9415da7df9feecc7170af" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36131 } }
I20260921 18:46:13.890477 9722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:13.891592 9728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "852aedb0dca9415da7df9feecc7170af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852aedb0dca9415da7df9feecc7170af" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36131 } } }
I20260921 18:46:13.891712 9728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:13.892040 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 852aedb0dca9415da7df9feecc7170af. Latest consensus state: current_term: 1 leader_uuid: "852aedb0dca9415da7df9feecc7170af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "852aedb0dca9415da7df9feecc7170af" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36131 } } }
I20260921 18:46:13.892141 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:13.892105 9736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:13.895411 9736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:13.900382 9736 catalog_manager.cc:1347] Generated new cluster ID: 162ca799dfa7473d87bb90a1e70dc0da
I20260921 18:46:13.900480 9736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:13.906636 9736 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:13.907684 9736 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:13.912621 9736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 852aedb0dca9415da7df9feecc7170af: Generated new TSK 0
I20260921 18:46:13.913272 9736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:13.916564 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--builtin_ntp_servers=127.3.36.148:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260921 18:46:14.025836 9746 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:14.026078 9746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:14.026098 9746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:14.026170 9746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:14.030238 9746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:14.030318 9746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:14.034946 9746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--never_fsync=true
--heap_profile_path=/tmp/kudu.9746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:14.035885 9746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:14.037140 9746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:14.046128 9751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.046134 9754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.046144 9752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:14.046504 9746 server_base.cc:1034] running on GCE node
I20260921 18:46:14.046904 9746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:14.047483 9746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:14.048660 9746 hybrid_clock.cc:648] HybridClock initialized: now 1790016374048635 us; error 47 us; skew 500 ppm
I20260921 18:46:14.050423 9746 webserver.cc:460] Webserver started at http://127.3.36.129:34089/ using document root <none> and password file <none>
I20260921 18:46:14.050958 9746 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:14.051044 9746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:14.051270 9746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:14.053059 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2948bd09bded4bf3acb2aff022f83f2e"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.053670 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2948bd09bded4bf3acb2aff022f83f2e"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.057056 9746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:14.059408 9760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.060482 9746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:14.060614 9746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2948bd09bded4bf3acb2aff022f83f2e"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.060716 9746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:14.074350 9746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:14.074935 9746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:14.075120 9746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:14.075665 9746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:14.076563 9746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:14.076632 9746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.076700 9746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:14.076750 9746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.084849 9746 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:35811
I20260921 18:46:14.084913 9873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:35811 every 8 connection(s)
I20260921 18:46:14.085760 9746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:14.092199 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9746
I20260921 18:46:14.092324 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:14.094749 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--builtin_ntp_servers=127.3.36.148:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:14.095855 9874 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36131
I20260921 18:46:14.096091 9874 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:14.096541 9874 heartbeater.cc:511] Master 127.3.36.190:36131 requested a full tablet report, sending...
I20260921 18:46:14.097916 9686 ts_manager.cc:194] Registered new tserver with Master: 2948bd09bded4bf3acb2aff022f83f2e (127.3.36.129:35811)
I20260921 18:46:14.099066 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:42149
W20260921 18:46:14.200279 9877 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:14.200536 9877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:14.200613 9877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:14.200685 9877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:14.204595 9877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:14.204725 9877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:14.209276 9877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--never_fsync=true
--heap_profile_path=/tmp/kudu.9877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:14.210215 9877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:14.211432 9877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:14.220109 9882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.220109 9885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.220141 9883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:14.220662 9877 server_base.cc:1034] running on GCE node
I20260921 18:46:14.221029 9877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:14.221661 9877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:14.222872 9877 hybrid_clock.cc:648] HybridClock initialized: now 1790016374222857 us; error 25 us; skew 500 ppm
I20260921 18:46:14.224509 9877 webserver.cc:460] Webserver started at http://127.3.36.130:38577/ using document root <none> and password file <none>
I20260921 18:46:14.225070 9877 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:14.225131 9877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:14.225364 9877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:14.227146 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "306f3f8e93c645d98e5918688eea0182"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.227686 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "306f3f8e93c645d98e5918688eea0182"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.231215 9877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:14.233459 9891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.234632 9877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:14.234742 9877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "306f3f8e93c645d98e5918688eea0182"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.234834 9877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:14.248471 9877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:14.249030 9877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:14.249198 9877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:14.249809 9877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:14.250665 9877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:14.250710 9877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.250774 9877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:14.250810 9877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.258925 9877 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:44331
I20260921 18:46:14.258946 10004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:44331 every 8 connection(s)
I20260921 18:46:14.259755 9877 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:14.260620 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 9877
I20260921 18:46:14.260737 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:14.263027 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--builtin_ntp_servers=127.3.36.148:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:14.271075 10005 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36131
I20260921 18:46:14.271334 10005 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:14.271981 10005 heartbeater.cc:511] Master 127.3.36.190:36131 requested a full tablet report, sending...
I20260921 18:46:14.273104 9686 ts_manager.cc:194] Registered new tserver with Master: 306f3f8e93c645d98e5918688eea0182 (127.3.36.130:44331)
I20260921 18:46:14.273717 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:52201
W20260921 18:46:14.372267 10008 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:14.372504 10008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:14.372524 10008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:14.372603 10008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:14.376516 10008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:14.376595 10008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:14.381088 10008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--never_fsync=true
--heap_profile_path=/tmp/kudu.10008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:14.382036 10008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:14.383335 10008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:14.392104 10016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.392119 10014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.392119 10013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:14.392297 10008 server_base.cc:1034] running on GCE node
I20260921 18:46:14.392743 10008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:14.393316 10008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:14.394485 10008 hybrid_clock.cc:648] HybridClock initialized: now 1790016374394475 us; error 26 us; skew 500 ppm
I20260921 18:46:14.396076 10008 webserver.cc:460] Webserver started at http://127.3.36.131:44345/ using document root <none> and password file <none>
I20260921 18:46:14.396548 10008 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:14.396610 10008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:14.396759 10008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:14.398499 10008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b7b9ad1939394aedaad34c62991cd077"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.399003 10008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b7b9ad1939394aedaad34c62991cd077"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.402438 10008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:14.404651 10022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.405869 10008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:14.406003 10008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b7b9ad1939394aedaad34c62991cd077"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.406109 10008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:14.431459 10008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:14.432117 10008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:14.432302 10008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:14.432874 10008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:14.433818 10008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:14.433887 10008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.433993 10008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:14.434047 10008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.442533 10008 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:36707
I20260921 18:46:14.442543 10135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:36707 every 8 connection(s)
I20260921 18:46:14.443499 10008 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:14.449065 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10008
I20260921 18:46:14.449203 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:14.451371 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.132:0
--local_ip_for_outbound_sockets=127.3.36.132
--webserver_interface=127.3.36.132
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--builtin_ntp_servers=127.3.36.148:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:14.453641 10136 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36131
I20260921 18:46:14.453873 10136 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:14.454330 10136 heartbeater.cc:511] Master 127.3.36.190:36131 requested a full tablet report, sending...
I20260921 18:46:14.455276 9686 ts_manager.cc:194] Registered new tserver with Master: b7b9ad1939394aedaad34c62991cd077 (127.3.36.131:36707)
I20260921 18:46:14.455817 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:36465
W20260921 18:46:14.565984 10139 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:14.566401 10139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:14.566453 10139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:14.566500 10139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:14.572419 10139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:14.572568 10139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.132
I20260921 18:46:14.577752 10139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.36.132
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--never_fsync=true
--heap_profile_path=/tmp/kudu.10139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.132
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:14.578758 10139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:14.580084 10139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:14.589340 10145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.589529 10147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.589349 10144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:14.589567 10139 server_base.cc:1034] running on GCE node
I20260921 18:46:14.590114 10139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:14.590674 10139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:14.591964 10139 hybrid_clock.cc:648] HybridClock initialized: now 1790016374591775 us; error 72 us; skew 500 ppm
I20260921 18:46:14.593657 10139 webserver.cc:460] Webserver started at http://127.3.36.132:35223/ using document root <none> and password file <none>
I20260921 18:46:14.594174 10139 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:14.594233 10139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:14.594463 10139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:14.596230 10139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ec0cbaee2a6445bc8e09a0f497429451"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.596760 10139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ec0cbaee2a6445bc8e09a0f497429451"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.600163 10139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260921 18:46:14.602396 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.603514 10139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:14.603634 10139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ec0cbaee2a6445bc8e09a0f497429451"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.603736 10139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:14.629539 10139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:14.630400 10139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:14.630623 10139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:14.631330 10139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:14.632628 10139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:14.632710 10139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.632772 10139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:14.632805 10139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.644841 10139 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.132:32929
I20260921 18:46:14.644860 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.132:32929 every 8 connection(s)
I20260921 18:46:14.646107 10139 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260921 18:46:14.647513 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10139
I20260921 18:46:14.647650 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260921 18:46:14.650341 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.133:0
--local_ip_for_outbound_sockets=127.3.36.133
--webserver_interface=127.3.36.133
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--builtin_ntp_servers=127.3.36.148:41123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:14.659574 10267 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36131
I20260921 18:46:14.659802 10267 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:14.660419 10267 heartbeater.cc:511] Master 127.3.36.190:36131 requested a full tablet report, sending...
I20260921 18:46:14.661579 9686 ts_manager.cc:194] Registered new tserver with Master: ec0cbaee2a6445bc8e09a0f497429451 (127.3.36.132:32929)
I20260921 18:46:14.662151 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.132:46411
W20260921 18:46:14.761405 10270 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:14.761705 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:14.761752 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:14.761812 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:14.765890 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:14.766026 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.133
I20260921 18:46:14.770558 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.3.36.133
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36131
--never_fsync=true
--heap_profile_path=/tmp/kudu.10270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.133
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:14.771498 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:14.772768 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:14.782146 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:14.782217 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:14.782182 10270 server_base.cc:1034] running on GCE node
W20260921 18:46:14.782191 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:14.782764 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:14.783352 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:14.784493 10270 hybrid_clock.cc:648] HybridClock initialized: now 1790016374784458 us; error 46 us; skew 500 ppm
I20260921 18:46:14.786312 10270 webserver.cc:460] Webserver started at http://127.3.36.133:44225/ using document root <none> and password file <none>
I20260921 18:46:14.786852 10270 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:14.786909 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:14.787056 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:14.788797 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9c51a1b7cec14335a67d10683ea51ea0"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.789350 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9c51a1b7cec14335a67d10683ea51ea0"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.792948 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260921 18:46:14.795307 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.796546 10270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:14.796669 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9c51a1b7cec14335a67d10683ea51ea0"
format_stamp: "Formatted at 2026-09-21 18:46:14 on dist-test-slave-l4st"
I20260921 18:46:14.796749 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:14.828843 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:14.829438 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:14.829634 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:14.830132 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:14.831019 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:14.831064 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.831102 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:14.831118 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:14.839355 10270 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.133:34643
I20260921 18:46:14.839382 10397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.133:34643 every 8 connection(s)
I20260921 18:46:14.840191 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260921 18:46:14.847397 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10270
I20260921 18:46:14.847517 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260921 18:46:14.849900 10398 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36131
I20260921 18:46:14.850118 10398 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:14.850569 10398 heartbeater.cc:511] Master 127.3.36.190:36131 requested a full tablet report, sending...
I20260921 18:46:14.851490 9686 ts_manager.cc:194] Registered new tserver with Master: 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133:34643)
I20260921 18:46:14.852018 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.133:48597
I20260921 18:46:14.861543 3218 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260921 18:46:14.870422 9686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37338:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260921 18:46:14.873179 9686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20260921 18:46:14.892277 10070 tablet_service.cc:1467] Processing CreateTablet for tablet 56af2bd9eb124089bf0ad96f99d31476 (DEFAULT_TABLE table=TestTable [id=bbbe0e03e81c4119aeccfe31aed7c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:14.892313 9808 tablet_service.cc:1467] Processing CreateTablet for tablet 56af2bd9eb124089bf0ad96f99d31476 (DEFAULT_TABLE table=TestTable [id=bbbe0e03e81c4119aeccfe31aed7c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:14.892617 10201 tablet_service.cc:1467] Processing CreateTablet for tablet 56af2bd9eb124089bf0ad96f99d31476 (DEFAULT_TABLE table=TestTable [id=bbbe0e03e81c4119aeccfe31aed7c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:14.893344 10070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56af2bd9eb124089bf0ad96f99d31476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:14.893343 9808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56af2bd9eb124089bf0ad96f99d31476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:14.893426 10201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56af2bd9eb124089bf0ad96f99d31476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:14.895282 10332 tablet_service.cc:1467] Processing CreateTablet for tablet 56af2bd9eb124089bf0ad96f99d31476 (DEFAULT_TABLE table=TestTable [id=bbbe0e03e81c4119aeccfe31aed7c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:14.896138 10332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56af2bd9eb124089bf0ad96f99d31476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:14.896927 9939 tablet_service.cc:1467] Processing CreateTablet for tablet 56af2bd9eb124089bf0ad96f99d31476 (DEFAULT_TABLE table=TestTable [id=bbbe0e03e81c4119aeccfe31aed7c269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:14.898360 9939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56af2bd9eb124089bf0ad96f99d31476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:14.901216 10420 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: Bootstrap starting.
I20260921 18:46:14.903205 10420 tablet_bootstrap.cc:654] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:14.903856 10420 log.cc:824] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:14.904500 10419 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: Bootstrap starting.
I20260921 18:46:14.904546 10422 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: Bootstrap starting.
I20260921 18:46:14.905673 10423 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: Bootstrap starting.
I20260921 18:46:14.906594 10422 tablet_bootstrap.cc:654] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:14.906634 10419 tablet_bootstrap.cc:654] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:14.907403 10419 log.cc:824] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:14.907919 10423 tablet_bootstrap.cc:654] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:14.907403 10422 log.cc:824] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:14.908792 10423 log.cc:824] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:14.909336 10425 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: Bootstrap starting.
I20260921 18:46:14.910297 10419 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: No bootstrap required, opened a new log
I20260921 18:46:14.910297 10422 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: No bootstrap required, opened a new log
I20260921 18:46:14.910470 10422 ts_tablet_manager.cc:1397] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260921 18:46:14.910470 10419 ts_tablet_manager.cc:1397] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260921 18:46:14.910643 10423 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: No bootstrap required, opened a new log
I20260921 18:46:14.910799 10423 ts_tablet_manager.cc:1397] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:14.910960 10420 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: No bootstrap required, opened a new log
I20260921 18:46:14.911115 10420 ts_tablet_manager.cc:1397] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260921 18:46:14.911336 10425 tablet_bootstrap.cc:654] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:14.912055 10425 log.cc:824] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:14.913897 10425 tablet_bootstrap.cc:492] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: No bootstrap required, opened a new log
I20260921 18:46:14.914065 10425 ts_tablet_manager.cc:1397] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:14.915146 10422 raft_consensus.cc:348] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.915205 10423 raft_consensus.cc:348] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.915400 10422 raft_consensus.cc:729] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 306f3f8e93c645d98e5918688eea0182, State: Initialized, Role: FOLLOWER
I20260921 18:46:14.915442 10423 raft_consensus.cc:729] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c51a1b7cec14335a67d10683ea51ea0, State: Initialized, Role: FOLLOWER
I20260921 18:46:14.915707 10422 consensus_queue.cc:260] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.915735 10423 consensus_queue.cc:260] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.916689 10422 ts_tablet_manager.cc:1428] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:46:14.916705 10423 ts_tablet_manager.cc:1428] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260921 18:46:14.917004 10005 heartbeater.cc:503] Master 127.3.36.190:36131 was elected leader, sending a full tablet report...
I20260921 18:46:14.917057 10398 heartbeater.cc:503] Master 127.3.36.190:36131 was elected leader, sending a full tablet report...
I20260921 18:46:14.918130 10425 raft_consensus.cc:348] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.918363 10425 raft_consensus.cc:729] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2948bd09bded4bf3acb2aff022f83f2e, State: Initialized, Role: FOLLOWER
I20260921 18:46:14.918278 10419 raft_consensus.cc:348] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.918497 10419 raft_consensus.cc:729] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b9ad1939394aedaad34c62991cd077, State: Initialized, Role: FOLLOWER
I20260921 18:46:14.918606 10425 consensus_queue.cc:260] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.918738 10419 consensus_queue.cc:260] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.919271 9874 heartbeater.cc:503] Master 127.3.36.190:36131 was elected leader, sending a full tablet report...
I20260921 18:46:14.919409 10425 ts_tablet_manager.cc:1428] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:46:14.919559 10419 ts_tablet_manager.cc:1428] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260921 18:46:14.920380 10136 heartbeater.cc:503] Master 127.3.36.190:36131 was elected leader, sending a full tablet report...
I20260921 18:46:14.922384 10420 raft_consensus.cc:348] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.922649 10420 raft_consensus.cc:729] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0cbaee2a6445bc8e09a0f497429451, State: Initialized, Role: FOLLOWER
I20260921 18:46:14.922932 10420 consensus_queue.cc:260] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.923859 10420 ts_tablet_manager.cc:1428] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260921 18:46:14.924136 10267 heartbeater.cc:503] Master 127.3.36.190:36131 was elected leader, sending a full tablet report...
I20260921 18:46:14.927376 3218 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20260921 18:46:14.928617 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2948bd09bded4bf3acb2aff022f83f2e to finish bootstrapping
I20260921 18:46:14.933646 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 306f3f8e93c645d98e5918688eea0182 to finish bootstrapping
I20260921 18:46:14.938154 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b7b9ad1939394aedaad34c62991cd077 to finish bootstrapping
I20260921 18:46:14.942713 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ec0cbaee2a6445bc8e09a0f497429451 to finish bootstrapping
W20260921 18:46:14.944595 10137 tablet.cc:2369] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:14.947628 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9c51a1b7cec14335a67d10683ea51ea0 to finish bootstrapping
I20260921 18:46:14.956312 9959 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476"
dest_uuid: "306f3f8e93c645d98e5918688eea0182"
from {username='slave'} at 127.0.0.1:54954
I20260921 18:46:14.956492 9959 raft_consensus.cc:482] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:14.956579 9959 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:14.957523 9959 raft_consensus.cc:504] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.958585 9959 leader_election.cc:288] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [CANDIDATE]: Term 1 election: Requested vote from peers 2948bd09bded4bf3acb2aff022f83f2e (127.3.36.129:35811), ec0cbaee2a6445bc8e09a0f497429451 (127.3.36.132:32929), b7b9ad1939394aedaad34c62991cd077 (127.3.36.131:36707), 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133:34643)
I20260921 18:46:14.963836 9828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "306f3f8e93c645d98e5918688eea0182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2948bd09bded4bf3acb2aff022f83f2e"
I20260921 18:46:14.964066 9828 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:14.964227 10221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "306f3f8e93c645d98e5918688eea0182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec0cbaee2a6445bc8e09a0f497429451"
I20260921 18:46:14.964329 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "306f3f8e93c645d98e5918688eea0182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7b9ad1939394aedaad34c62991cd077"
I20260921 18:46:14.964411 10221 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:14.964495 10090 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:14.965332 9828 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306f3f8e93c645d98e5918688eea0182 in term 1.
I20260921 18:46:14.965463 10352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "306f3f8e93c645d98e5918688eea0182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c51a1b7cec14335a67d10683ea51ea0"
I20260921 18:46:14.965677 10352 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:14.965723 10221 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306f3f8e93c645d98e5918688eea0182 in term 1.
I20260921 18:46:14.965752 10090 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306f3f8e93c645d98e5918688eea0182 in term 1.
I20260921 18:46:14.966131 9892 leader_election.cc:302] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2948bd09bded4bf3acb2aff022f83f2e, 306f3f8e93c645d98e5918688eea0182, ec0cbaee2a6445bc8e09a0f497429451; no voters:
I20260921 18:46:14.966437 10430 raft_consensus.cc:2781] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:14.966637 10430 raft_consensus.cc:686] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 LEADER]: Becoming Leader. State: Replica: 306f3f8e93c645d98e5918688eea0182, State: Running, Role: LEADER
I20260921 18:46:14.966892 10352 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 306f3f8e93c645d98e5918688eea0182 in term 1.
I20260921 18:46:14.966957 10430 consensus_queue.cc:237] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:14.970160 9684 catalog_manager.cc:5495] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 reported cstate change: term changed from 0 to 1, leader changed from <none> to 306f3f8e93c645d98e5918688eea0182 (127.3.36.130). New cstate: current_term: 1 leader_uuid: "306f3f8e93c645d98e5918688eea0182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } health_report { overall_health: HEALTHY } } }
W20260921 18:46:15.011058 10006 tablet.cc:2369] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:46:15.087205 9875 tablet.cc:2369] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:46:15.091063 10399 tablet.cc:2369] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:46:15.147284 10268 tablet.cc:2369] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:15.406684 10430 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:15.468245 10430 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:15.474538 10430 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:15.475112 10430 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:15.478251 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9655
W20260921 18:46:15.484591 10267 heartbeater.cc:644] Failed to heartbeat to 127.3.36.190:36131 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.3.36.190:36131 (error 108)
W20260921 18:46:15.484611 10026 connection.cc:535] client connection to 127.3.36.190:36131 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260921 18:46:15.484936 10136 heartbeater.cc:644] Failed to heartbeat to 127.3.36.190:36131 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260921 18:46:15.485626 3218 raft_consensus_election-itest.cc:666] Removing tserver with uuid 9c51a1b7cec14335a67d10683ea51ea0
W20260921 18:46:15.486194 10459 negotiation.cc:337] Failed RPC negotiation. Trace:
0921 18:46:15.485310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.36.190:36131 (local address 127.3.36.131:34745)
0921 18:46:15.485543 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0921 18:46:15.485615 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.36.190:36131: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
W20260921 18:46:15.486303 10458 negotiation.cc:337] Failed RPC negotiation. Trace:
0921 18:46:15.485083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.36.190:36131 (local address 127.3.36.132:50521)
0921 18:46:15.485302 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0921 18:46:15.485386 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.36.190:36131: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260921 18:46:15.486838 9959 consensus_queue.cc:237] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }
I20260921 18:46:15.488101 10090 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:15.488181 9828 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:15.488405 10221 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:15.488484 10445 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:46:15.488662 10430 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:46:15.489100 10430 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:46:15.490327 10445 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } }
I20260921 18:46:15.490707 9828 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } }
I20260921 18:46:15.491190 10221 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } }
I20260921 18:46:15.492610 10090 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } }
W20260921 18:46:15.495033 9874 heartbeater.cc:644] Failed to heartbeat to 127.3.36.190:36131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.36.190:36131: connect: Connection refused (error 111)
W20260921 18:46:15.495102 10005 heartbeater.cc:644] Failed to heartbeat to 127.3.36.190:36131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.36.190:36131: connect: Connection refused (error 111)
I20260921 18:46:15.508199 3218 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20260921 18:46:15.539703 3218 raft_consensus_election-itest.cc:684] Adding back Peer 9c51a1b7cec14335a67d10683ea51ea0 and expecting timeout...
I20260921 18:46:15.540844 9959 consensus_queue.cc:237] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } }
I20260921 18:46:15.542411 10352 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260921 18:46:15.542814 10448 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:46:15.544262 10352 raft_consensus.cc:858] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } }
W20260921 18:46:15.544350 10352 raft_consensus.cc:1565] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } }
I20260921 18:46:15.544660 10352 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } }
W20260921 18:46:15.547175 10398 heartbeater.cc:644] Failed to heartbeat to 127.3.36.190:36131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.36.190:36131: connect: Connection refused (error 111)
I20260921 18:46:15.547227 10352 raft_consensus.cc:1467] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260921 18:46:15.640832 3218 raft_consensus_election-itest.cc:688] Timeout achieved.
I20260921 18:46:15.646076 3218 raft_consensus_election-itest.cc:699] Forcing Peer 306f3f8e93c645d98e5918688eea0182 to step down...
I20260921 18:46:15.646554 9959 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476"
dest_uuid: "306f3f8e93c645d98e5918688eea0182"
from {username='slave'} at 127.0.0.1:54954
I20260921 18:46:15.646736 9959 raft_consensus.cc:580] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 LEADER]: Received request to step down
I20260921 18:46:15.646791 9959 raft_consensus.cc:3032] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:15.646862 9959 raft_consensus.cc:729] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 306f3f8e93c645d98e5918688eea0182, State: Running, Role: LEADER
I20260921 18:46:15.647033 9959 consensus_queue.cc:260] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } }
I20260921 18:46:15.647218 9959 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:15.647586 3218 raft_consensus_election-itest.cc:705] Resuming Peer 2948bd09bded4bf3acb2aff022f83f2e ...
I20260921 18:46:15.648167 3218 raft_consensus_election-itest.cc:710] Trying to elect Peer 9c51a1b7cec14335a67d10683ea51ea0 ...
I20260921 18:46:15.648618 10352 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476"
dest_uuid: "9c51a1b7cec14335a67d10683ea51ea0"
from {username='slave'} at 127.0.0.1:53814
I20260921 18:46:15.648696 9828 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260921 18:46:15.648805 10352 raft_consensus.cc:482] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:15.648875 10352 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:15.651377 10352 raft_consensus.cc:504] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } }
I20260921 18:46:15.652545 10352 leader_election.cc:288] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [CANDIDATE]: Term 2 election: Requested vote from peers 2948bd09bded4bf3acb2aff022f83f2e (127.3.36.129:35811), ec0cbaee2a6445bc8e09a0f497429451 (127.3.36.132:32929), b7b9ad1939394aedaad34c62991cd077 (127.3.36.131:36707), 306f3f8e93c645d98e5918688eea0182 (127.3.36.130:44331)
I20260921 18:46:15.654830 3218 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20260921 18:46:15.657196 9828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "9c51a1b7cec14335a67d10683ea51ea0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2948bd09bded4bf3acb2aff022f83f2e"
I20260921 18:46:15.657322 9828 raft_consensus.cc:1799] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 1 FOLLOWER]: Handling vote request from an unknown peer 9c51a1b7cec14335a67d10683ea51ea0
I20260921 18:46:15.657380 9828 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:15.657999 9959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "9c51a1b7cec14335a67d10683ea51ea0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "306f3f8e93c645d98e5918688eea0182"
I20260921 18:46:15.658416 9828 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c51a1b7cec14335a67d10683ea51ea0 in term 2.
I20260921 18:46:15.659024 9959 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c51a1b7cec14335a67d10683ea51ea0 in term 2.
I20260921 18:46:15.659361 10287 leader_election.cc:302] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2948bd09bded4bf3acb2aff022f83f2e, 306f3f8e93c645d98e5918688eea0182, 9c51a1b7cec14335a67d10683ea51ea0; no voters:
I20260921 18:46:15.659567 10429 raft_consensus.cc:2781] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 2 FOLLOWER]: Leader election won for term 2
I20260921 18:46:15.659781 10429 raft_consensus.cc:686] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 2 LEADER]: Becoming Leader. State: Replica: 9c51a1b7cec14335a67d10683ea51ea0, State: Running, Role: LEADER
I20260921 18:46:15.660079 10429 consensus_queue.cc:237] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } }
I20260921 18:46:15.757637 3218 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260921 18:46:15.960395 3218 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20260921 18:46:16.062523 9959 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 2 FOLLOWER]: Refusing update from remote peer 9c51a1b7cec14335a67d10683ea51ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260921 18:46:16.063158 10429 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260921 18:46:16.212908 9828 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 2 FOLLOWER]: Refusing update from remote peer 9c51a1b7cec14335a67d10683ea51ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260921 18:46:16.213392 10429 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260921 18:46:16.215502 10475 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } } }
I20260921 18:46:16.215884 9828 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P 2948bd09bded4bf3acb2aff022f83f2e [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } } }
I20260921 18:46:16.217880 9959 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P 306f3f8e93c645d98e5918688eea0182 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } } }
W20260921 18:46:16.219179 9959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:54954 (request call id 52) took 678 ms (client timeout 100 ms). Trace:
W20260921 18:46:16.219301 9959 rpcz_store.cc:269] 0921 18:46:15.540327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:15.540399 (+ 72us) service_pool.cc:227] Handling call
0921 18:46:15.541569 (+ 1170us) log.cc:842] Serialized 600 byte log entry
0921 18:46:16.219144 (+677575us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44,"raft.queue_time_us":418,"raft.run_cpu_time_us":456,"raft.run_wall_time_us":455,"spinlock_wait_cycles":139008}
I20260921 18:46:16.263137 3218 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20260921 18:46:16.265321 10221 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260921 18:46:16.269959 10090 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 1 FOLLOWER]: Refusing update from remote peer 306f3f8e93c645d98e5918688eea0182: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260921 18:46:16.275058 10090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "9c51a1b7cec14335a67d10683ea51ea0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b7b9ad1939394aedaad34c62991cd077"
I20260921 18:46:16.275179 10090 raft_consensus.cc:1799] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 1 FOLLOWER]: Handling vote request from an unknown peer 9c51a1b7cec14335a67d10683ea51ea0
I20260921 18:46:16.275233 10090 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:16.276285 10090 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c51a1b7cec14335a67d10683ea51ea0 in term 2.
I20260921 18:46:16.276696 10089 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 2 FOLLOWER]: Refusing update from remote peer 9c51a1b7cec14335a67d10683ea51ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260921 18:46:16.277063 10429 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260921 18:46:16.277405 10220 raft_consensus.cc:3037] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:16.278615 10220 raft_consensus.cc:1261] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 2 FOLLOWER]: Refusing update from remote peer 9c51a1b7cec14335a67d10683ea51ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260921 18:46:16.278851 10221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56af2bd9eb124089bf0ad96f99d31476" candidate_uuid: "9c51a1b7cec14335a67d10683ea51ea0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ec0cbaee2a6445bc8e09a0f497429451"
I20260921 18:46:16.278970 10221 raft_consensus.cc:1799] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 2 FOLLOWER]: Handling vote request from an unknown peer 9c51a1b7cec14335a67d10683ea51ea0
I20260921 18:46:16.279294 10429 consensus_queue.cc:1035] T 56af2bd9eb124089bf0ad96f99d31476 P 9c51a1b7cec14335a67d10683ea51ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260921 18:46:16.279808 10221 raft_consensus.cc:2454] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c51a1b7cec14335a67d10683ea51ea0 in term 2.
I20260921 18:46:16.279692 10089 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P b7b9ad1939394aedaad34c62991cd077 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } } }
I20260921 18:46:16.282104 10221 raft_consensus.cc:2932] T 56af2bd9eb124089bf0ad96f99d31476 P ec0cbaee2a6445bc8e09a0f497429451 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9c51a1b7cec14335a67d10683ea51ea0 (127.3.36.133) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "2948bd09bded4bf3acb2aff022f83f2e" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 35811 } } peers { permanent_uuid: "ec0cbaee2a6445bc8e09a0f497429451" member_type: VOTER last_known_addr { host: "127.3.36.132" port: 32929 } } peers { permanent_uuid: "b7b9ad1939394aedaad34c62991cd077" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 36707 } } peers { permanent_uuid: "306f3f8e93c645d98e5918688eea0182" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 44331 } } peers { permanent_uuid: "9c51a1b7cec14335a67d10683ea51ea0" member_type: VOTER last_known_addr { host: "127.3.36.133" port: 34643 } attrs { } } }
I20260921 18:46:16.283052 10491 mvcc.cc:205] Tried to move back new op lower bound from 7331907077193895936 to 7331907074707677184. Current Snapshot: MvccSnapshot[applied={T|T < 7331907077193895936}]
I20260921 18:46:16.284188 3218 cluster_itest_util.cc:256] Not converged past 5 yet: 2.5 2.5 1.2 2.5 2.5
I20260921 18:46:16.286968 10495 mvcc.cc:205] Tried to move back new op lower bound from 7331907077193895936 to 7331907074707677184. Current Snapshot: MvccSnapshot[applied={T|T < 7331907077193895936}]
I20260921 18:46:16.388423 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9746
I20260921 18:46:16.394291 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 9877
I20260921 18:46:16.400130 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 10008
I20260921 18:46:16.405182 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 10139
I20260921 18:46:16.410388 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 10270
2026-09-21T18:46:16Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (2740 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20260921 18:46:16.432080 3218 test_util.cc:268] Using random seed: -1438489141
I20260921 18:46:16.432801 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:16.432835 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:16.432847 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:46:16.432857 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:46:16.432868 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:16Z Disabled control of system clock
I20260921 18:46:16.441757 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36491
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:36491
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260921 18:46:16.548328 10506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:16.548590 10506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:16.548653 10506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:16.552727 10506 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:16.552812 10506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:16.552853 10506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:16.552907 10506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:16.552959 10506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:16.557664 10506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:36491
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:16.558655 10506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:16.559974 10506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:16.567970 10511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:16.568063 10512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:16.568171 10506 server_base.cc:1034] running on GCE node
W20260921 18:46:16.568379 10514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:16.568912 10506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:16.569941 10506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:16.571094 10506 hybrid_clock.cc:648] HybridClock initialized: now 1790016376571075 us; error 28 us; skew 500 ppm
I20260921 18:46:16.572846 10506 webserver.cc:460] Webserver started at http://127.3.36.190:33289/ using document root <none> and password file <none>
I20260921 18:46:16.573395 10506 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:16.573451 10506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:16.573731 10506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:16.575416 10506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a699403814644b29a2dd1e8c4a5238b7"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:16.575941 10506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a699403814644b29a2dd1e8c4a5238b7"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:16.579377 10506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260921 18:46:16.581576 10520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:16.582747 10506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:46:16.582857 10506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a699403814644b29a2dd1e8c4a5238b7"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:16.582947 10506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:16.598153 10506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:16.598738 10506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:16.598919 10506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:16.605611 10506 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:36491
I20260921 18:46:16.605664 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:36491 every 8 connection(s)
I20260921 18:46:16.606539 10506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:16.609459 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:16.614176 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Bootstrap starting.
I20260921 18:46:16.616393 10573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:16.616642 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10506
I20260921 18:46:16.616726 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:16.617172 10573 log.cc:824] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:16.619143 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: No bootstrap required, opened a new log
I20260921 18:46:16.622915 10573 raft_consensus.cc:348] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:16.623150 10573 raft_consensus.cc:374] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:16.623209 10573 raft_consensus.cc:729] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a699403814644b29a2dd1e8c4a5238b7, State: Initialized, Role: FOLLOWER
I20260921 18:46:16.623656 10573 consensus_queue.cc:260] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:16.623811 10573 raft_consensus.cc:388] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:16.623876 10573 raft_consensus.cc:482] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:16.623989 10573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:16.624760 10573 raft_consensus.cc:504] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:16.625110 10573 leader_election.cc:302] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a699403814644b29a2dd1e8c4a5238b7; no voters:
I20260921 18:46:16.625407 10573 leader_election.cc:288] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:16.625527 10578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:16.625782 10578 raft_consensus.cc:686] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 LEADER]: Becoming Leader. State: Replica: a699403814644b29a2dd1e8c4a5238b7, State: Running, Role: LEADER
I20260921 18:46:16.626076 10578 consensus_queue.cc:237] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:16.626612 10573 sys_catalog.cc:564] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:16.627504 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a699403814644b29a2dd1e8c4a5238b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } } }
I20260921 18:46:16.627620 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:16.628031 10580 sys_catalog.cc:455] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a699403814644b29a2dd1e8c4a5238b7. Latest consensus state: current_term: 1 leader_uuid: "a699403814644b29a2dd1e8c4a5238b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } } }
I20260921 18:46:16.628103 10587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:16.628189 10580 sys_catalog.cc:458] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:16.631068 10587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:16.636720 10587 catalog_manager.cc:1347] Generated new cluster ID: 00b75e503eb24ca6b8a91ae55361fd2c
I20260921 18:46:16.636790 10587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:16.655407 10587 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:16.656275 10587 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:16.667299 10587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Generated new TSK 0
I20260921 18:46:16.667996 10587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:16.678148 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36491
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260921 18:46:16.792246 10597 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260921 18:46:16.792572 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:16.792627 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:16.792690 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:16.796742 10597 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260921 18:46:16.796821 10597 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260921 18:46:16.796864 10597 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:16.796912 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:16.797168 10597 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260921 18:46:16.797257 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:16.802582 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36491
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:16.803732 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:16.805131 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:16.814536 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:16.814553 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:16.814549 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:16.814810 10597 server_base.cc:1034] running on GCE node
I20260921 18:46:16.815253 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:16.815814 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:16.816989 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790016376816961 us; error 52 us; skew 500 ppm
I20260921 18:46:16.818709 10597 webserver.cc:460] Webserver started at http://127.3.36.129:44629/ using document root <none> and password file <none>
I20260921 18:46:16.819247 10597 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:16.819335 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:16.819566 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:16.821326 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e11ca314edf44f8c84e10e4f0d82ce90"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:16.821925 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e11ca314edf44f8c84e10e4f0d82ce90"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:16.825220 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260921 18:46:16.827656 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:16.828894 10597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:16.829015 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e11ca314edf44f8c84e10e4f0d82ce90"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:16.829120 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:16.846720 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:16.847297 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:16.847476 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:16.848040 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:16.848870 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:16.848940 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:16.849004 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:16.849056 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:16.857787 10597 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:42703
I20260921 18:46:16.857817 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:42703 every 8 connection(s)
I20260921 18:46:16.858608 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:16.864673 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10597
I20260921 18:46:16.864794 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:16.866760 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36491
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260921 18:46:16.868635 10725 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36491
I20260921 18:46:16.868835 10725 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:16.869208 10725 heartbeater.cc:511] Master 127.3.36.190:36491 requested a full tablet report, sending...
I20260921 18:46:16.870558 10537 ts_manager.cc:194] Registered new tserver with Master: e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
I20260921 18:46:16.871585 10537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:44623
W20260921 18:46:16.974558 10728 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260921 18:46:16.974799 10728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:16.974818 10728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:16.974848 10728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:16.978777 10728 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260921 18:46:16.978821 10728 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260921 18:46:16.978843 10728 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:16.978894 10728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:16.978981 10728 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260921 18:46:16.979022 10728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:16.983523 10728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36491
--never_fsync=true
--heap_profile_path=/tmp/kudu.10728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:16.984454 10728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:16.985783 10728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:16.994755 10734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:16.994783 10733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:16.994786 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:16.994952 10728 server_base.cc:1034] running on GCE node
I20260921 18:46:16.995375 10728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:16.995945 10728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:16.997141 10728 hybrid_clock.cc:648] HybridClock initialized: now 1790016376997123 us; error 28 us; skew 500 ppm
I20260921 18:46:16.998855 10728 webserver.cc:460] Webserver started at http://127.3.36.130:34823/ using document root <none> and password file <none>
I20260921 18:46:16.999423 10728 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:16.999514 10728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:16.999730 10728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:17.001600 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2af2bc0080434cfeadd826373d56ccc5"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:17.002156 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2af2bc0080434cfeadd826373d56ccc5"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:17.005700 10728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:17.007920 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:17.009066 10728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:17.009195 10728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2af2bc0080434cfeadd826373d56ccc5"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:17.009300 10728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:17.020663 10728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:17.021247 10728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:17.021431 10728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:17.021981 10728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:17.022816 10728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:17.022895 10728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:17.022981 10728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:17.023037 10728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:17.032091 10728 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:40031
I20260921 18:46:17.032135 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:40031 every 8 connection(s)
I20260921 18:46:17.032972 10728 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:17.042443 10856 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36491
I20260921 18:46:17.042671 10856 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:17.042985 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10728
I20260921 18:46:17.043054 10856 heartbeater.cc:511] Master 127.3.36.190:36491 requested a full tablet report, sending...
I20260921 18:46:17.043147 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:17.043984 10537 ts_manager.cc:194] Registered new tserver with Master: 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031)
I20260921 18:46:17.044580 10537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:59375
I20260921 18:46:17.045306 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36491
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20260921 18:46:17.149688 10859 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260921 18:46:17.149937 10859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:17.149997 10859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:17.150063 10859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:17.153968 10859 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260921 18:46:17.154017 10859 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260921 18:46:17.154039 10859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:17.154095 10859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:17.154183 10859 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260921 18:46:17.154227 10859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:17.158815 10859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36491
--never_fsync=true
--heap_profile_path=/tmp/kudu.10859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:17.159766 10859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:17.161057 10859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:17.170158 10864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:17.170221 10867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:17.170325 10859 server_base.cc:1034] running on GCE node
W20260921 18:46:17.170153 10865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:17.170847 10859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:17.171397 10859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:17.172552 10859 hybrid_clock.cc:648] HybridClock initialized: now 1790016377172540 us; error 30 us; skew 500 ppm
I20260921 18:46:17.174401 10859 webserver.cc:460] Webserver started at http://127.3.36.131:42319/ using document root <none> and password file <none>
I20260921 18:46:17.174909 10859 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:17.174968 10859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:17.175115 10859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:17.176817 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fee1d4fe94924b07a24fc6a7acfd8365"
format_stamp: "Formatted at 2026-09-21 18:46:17 on dist-test-slave-l4st"
I20260921 18:46:17.177284 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fee1d4fe94924b07a24fc6a7acfd8365"
format_stamp: "Formatted at 2026-09-21 18:46:17 on dist-test-slave-l4st"
I20260921 18:46:17.180598 10859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:17.182744 10873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:17.183838 10859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:17.183964 10859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fee1d4fe94924b07a24fc6a7acfd8365"
format_stamp: "Formatted at 2026-09-21 18:46:17 on dist-test-slave-l4st"
I20260921 18:46:17.184063 10859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:17.217032 10859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:17.217638 10859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:17.217833 10859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:17.218364 10859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:17.219246 10859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:17.219309 10859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:17.219372 10859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:17.219424 10859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:17.227845 10859 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:44211
I20260921 18:46:17.227869 10986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:44211 every 8 connection(s)
I20260921 18:46:17.228708 10859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:17.230687 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 10859
I20260921 18:46:17.230810 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:17.238562 10987 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36491
I20260921 18:46:17.238737 10987 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:17.239053 10987 heartbeater.cc:511] Master 127.3.36.190:36491 requested a full tablet report, sending...
I20260921 18:46:17.239835 10537 ts_manager.cc:194] Registered new tserver with Master: fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211)
I20260921 18:46:17.240319 10537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:35613
I20260921 18:46:17.244545 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:17.252707 10537 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20260921 18:46:17.271127 10790 tablet_service.cc:1467] Processing CreateTablet for tablet 813af822f7214169aed018a62884703e (DEFAULT_TABLE table=TestTable [id=ed2d50e88d6e44ed9459c7307be5c1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:17.271534 10659 tablet_service.cc:1467] Processing CreateTablet for tablet 813af822f7214169aed018a62884703e (DEFAULT_TABLE table=TestTable [id=ed2d50e88d6e44ed9459c7307be5c1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:17.271530 10921 tablet_service.cc:1467] Processing CreateTablet for tablet 813af822f7214169aed018a62884703e (DEFAULT_TABLE table=TestTable [id=ed2d50e88d6e44ed9459c7307be5c1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:17.272061 10790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813af822f7214169aed018a62884703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:17.272457 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813af822f7214169aed018a62884703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:17.272457 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813af822f7214169aed018a62884703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:17.277299 11004 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Bootstrap starting.
I20260921 18:46:17.279048 11006 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Bootstrap starting.
I20260921 18:46:17.279044 11005 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Bootstrap starting.
I20260921 18:46:17.279961 11004 tablet_bootstrap.cc:654] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:17.280637 11004 log.cc:824] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:17.281060 11006 tablet_bootstrap.cc:654] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:17.281060 11005 tablet_bootstrap.cc:654] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:17.281880 11005 log.cc:824] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:17.281880 11006 log.cc:824] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:17.282297 11004 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: No bootstrap required, opened a new log
I20260921 18:46:17.282482 11004 ts_tablet_manager.cc:1397] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:17.283603 11006 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: No bootstrap required, opened a new log
I20260921 18:46:17.283598 11005 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: No bootstrap required, opened a new log
I20260921 18:46:17.283769 11005 ts_tablet_manager.cc:1397] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:46:17.283808 11006 ts_tablet_manager.cc:1397] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:17.285929 11004 raft_consensus.cc:348] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.286155 11004 raft_consensus.cc:374] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:17.286218 11004 raft_consensus.cc:729] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2bc0080434cfeadd826373d56ccc5, State: Initialized, Role: FOLLOWER
I20260921 18:46:17.286649 11004 consensus_queue.cc:260] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.287287 10856 heartbeater.cc:503] Master 127.3.36.190:36491 was elected leader, sending a full tablet report...
I20260921 18:46:17.287635 11004 ts_tablet_manager.cc:1428] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260921 18:46:17.288151 11006 raft_consensus.cc:348] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.288151 11005 raft_consensus.cc:348] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.288386 11006 raft_consensus.cc:374] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:17.288386 11005 raft_consensus.cc:374] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:17.288462 11005 raft_consensus.cc:729] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11ca314edf44f8c84e10e4f0d82ce90, State: Initialized, Role: FOLLOWER
I20260921 18:46:17.288478 11006 raft_consensus.cc:729] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee1d4fe94924b07a24fc6a7acfd8365, State: Initialized, Role: FOLLOWER
I20260921 18:46:17.288889 11006 consensus_queue.cc:260] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.288889 11005 consensus_queue.cc:260] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.289798 11005 ts_tablet_manager.cc:1428] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260921 18:46:17.289888 10987 heartbeater.cc:503] Master 127.3.36.190:36491 was elected leader, sending a full tablet report...
I20260921 18:46:17.290097 10725 heartbeater.cc:503] Master 127.3.36.190:36491 was elected leader, sending a full tablet report...
I20260921 18:46:17.290822 11006 ts_tablet_manager.cc:1428] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
W20260921 18:46:17.329672 10988 tablet.cc:2369] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:46:17.334198 10857 tablet.cc:2369] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260921 18:46:17.360219 10726 tablet.cc:2369] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:17.522728 11011 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:46:17.522969 11011 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.524130 11011 leader_election.cc:288] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031), fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211)
I20260921 18:46:17.528741 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2bc0080434cfeadd826373d56ccc5" is_pre_election: true
I20260921 18:46:17.528806 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" is_pre_election: true
I20260921 18:46:17.529088 10810 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11ca314edf44f8c84e10e4f0d82ce90 in term 0.
I20260921 18:46:17.529107 10941 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e11ca314edf44f8c84e10e4f0d82ce90 in term 0.
I20260921 18:46:17.529616 10614 leader_election.cc:302] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2bc0080434cfeadd826373d56ccc5, e11ca314edf44f8c84e10e4f0d82ce90; no voters:
I20260921 18:46:17.529893 11011 raft_consensus.cc:2781] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:46:17.529975 11011 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:46:17.530018 11011 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:17.530959 11011 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.531296 11011 leader_election.cc:288] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [CANDIDATE]: Term 1 election: Requested vote from peers 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031), fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211)
I20260921 18:46:17.531589 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2af2bc0080434cfeadd826373d56ccc5"
I20260921 18:46:17.531654 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fee1d4fe94924b07a24fc6a7acfd8365"
I20260921 18:46:17.531728 10810 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:17.531785 10941 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:17.532788 10810 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11ca314edf44f8c84e10e4f0d82ce90 in term 1.
I20260921 18:46:17.532847 10941 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e11ca314edf44f8c84e10e4f0d82ce90 in term 1.
I20260921 18:46:17.533079 10614 leader_election.cc:302] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2bc0080434cfeadd826373d56ccc5, e11ca314edf44f8c84e10e4f0d82ce90; no voters:
I20260921 18:46:17.533299 11011 raft_consensus.cc:2781] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:17.533537 11011 raft_consensus.cc:686] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 1 LEADER]: Becoming Leader. State: Replica: e11ca314edf44f8c84e10e4f0d82ce90, State: Running, Role: LEADER
I20260921 18:46:17.533859 11011 consensus_queue.cc:237] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:17.536635 10534 catalog_manager.cc:5495] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } } }
I20260921 18:46:17.552762 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:17.554580 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e11ca314edf44f8c84e10e4f0d82ce90 to finish bootstrapping
I20260921 18:46:17.561661 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2af2bc0080434cfeadd826373d56ccc5 to finish bootstrapping
I20260921 18:46:17.566449 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fee1d4fe94924b07a24fc6a7acfd8365 to finish bootstrapping
I20260921 18:46:17.583240 3218 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260921 18:46:17.685828 3218 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260921 18:46:17.888468 3218 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260921 18:46:17.959326 11011 consensus_queue.cc:1035] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:17.964665 11011 consensus_queue.cc:1035] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:18.303876 3218 test_util.cc:268] Using random seed: -1436617351
I20260921 18:46:18.609880 3218 test_util.cc:268] Using random seed: -1436311344
I20260921 18:46:19.014931 3218 test_util.cc:268] Using random seed: -1435906295
I20260921 18:46:19.464751 11026 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e11ca314edf44f8c84e10e4f0d82ce90)
I20260921 18:46:19.464924 11026 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:19.466044 11026 leader_election.cc:288] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211), e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
I20260921 18:46:19.466504 11028 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e11ca314edf44f8c84e10e4f0d82ce90)
I20260921 18:46:19.466675 11028 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:19.467811 11028 leader_election.cc:288] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031), e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
I20260921 18:46:19.470467 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "2af2bc0080434cfeadd826373d56ccc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" is_pre_election: true
I20260921 18:46:19.470587 10941 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af2bc0080434cfeadd826373d56ccc5 in term 1.
I20260921 18:46:19.470937 10743 leader_election.cc:302] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2bc0080434cfeadd826373d56ccc5, fee1d4fe94924b07a24fc6a7acfd8365; no voters:
I20260921 18:46:19.471169 11026 raft_consensus.cc:2781] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260921 18:46:19.471236 11026 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e11ca314edf44f8c84e10e4f0d82ce90)
I20260921 18:46:19.471276 11026 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:19.472172 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2af2bc0080434cfeadd826373d56ccc5" is_pre_election: true
I20260921 18:46:19.472471 11026 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:19.472678 10810 raft_consensus.cc:2379] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fee1d4fe94924b07a24fc6a7acfd8365 in current term 2: Already voted for candidate 2af2bc0080434cfeadd826373d56ccc5 in this term.
I20260921 18:46:19.472797 11026 leader_election.cc:288] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 2 election: Requested vote from peers fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211), e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
I20260921 18:46:19.473053 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "2af2bc0080434cfeadd826373d56ccc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fee1d4fe94924b07a24fc6a7acfd8365"
I20260921 18:46:19.473198 10941 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:19.474028 10941 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af2bc0080434cfeadd826373d56ccc5 in term 2.
I20260921 18:46:19.474309 10743 leader_election.cc:302] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2af2bc0080434cfeadd826373d56ccc5, fee1d4fe94924b07a24fc6a7acfd8365; no voters:
I20260921 18:46:19.474467 11026 raft_consensus.cc:2781] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260921 18:46:19.474630 11026 raft_consensus.cc:686] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 LEADER]: Becoming Leader. State: Replica: 2af2bc0080434cfeadd826373d56ccc5, State: Running, Role: LEADER
I20260921 18:46:19.474916 11026 consensus_queue.cc:237] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:19.477101 10534 catalog_manager.cc:5495] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 reported cstate change: term changed from 1 to 2, leader changed from e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) to 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130). New cstate: current_term: 2 leader_uuid: "2af2bc0080434cfeadd826373d56ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } } }
I20260921 18:46:19.520067 3218 test_util.cc:268] Using random seed: -1435401158
I20260921 18:46:19.623174 3218 test_util.cc:268] Using random seed: -1435298051
I20260921 18:46:19.634639 3218 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
I20260921 18:46:19.634944 3218 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20260921 18:46:19.648725 10941 raft_consensus.cc:1261] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Refusing update from remote peer 2af2bc0080434cfeadd826373d56ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260921 18:46:19.649266 11026 consensus_queue.cc:1035] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:46:19.655696 11051 mvcc.cc:205] Tried to move back new op lower bound from 7331907091040718848 to 7331907090331299840. Current Snapshot: MvccSnapshot[applied={T|T < 7331907091035447296}]
I20260921 18:46:19.720798 3218 raft_consensus-itest-base.cc:274] Waiting for log GC on 2af2bc0080434cfeadd826373d56ccc5
I20260921 18:46:19.737337 10988 maintenance_manager.cc:419] P fee1d4fe94924b07a24fc6a7acfd8365: Scheduling FlushMRSOp(813af822f7214169aed018a62884703e): 4207767 bytes log retention, and flush 6119689 bytes memory
I20260921 18:46:19.741237 10857 maintenance_manager.cc:419] P 2af2bc0080434cfeadd826373d56ccc5: Scheduling FlushMRSOp(813af822f7214169aed018a62884703e): 4602274 bytes log retention, and flush 6119689 bytes memory
I20260921 18:46:19.805210 10878 maintenance_manager.cc:643] P fee1d4fe94924b07a24fc6a7acfd8365: FlushMRSOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.067s user 0.060s sys 0.004s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":206,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":666,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":39,"thread_start_us":252,"threads_started":2}
I20260921 18:46:19.806186 10747 maintenance_manager.cc:643] P 2af2bc0080434cfeadd826373d56ccc5: FlushMRSOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.064s user 0.064s sys 0.000s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":2796,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":654,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":39,"thread_start_us":191,"threads_started":2}
I20260921 18:46:19.806596 10988 maintenance_manager.cc:419] P fee1d4fe94924b07a24fc6a7acfd8365: Scheduling LogGCOp(813af822f7214169aed018a62884703e): free 2629823 bytes of WAL
I20260921 18:46:19.806849 10878 log_reader.cc:386] T 813af822f7214169aed018a62884703e: removed 3 log segments from log reader
I20260921 18:46:19.806980 10878 log.cc:1076] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/wals/813af822f7214169aed018a62884703e/wal-000000001 (ops 1-8)
I20260921 18:46:19.807165 10878 log.cc:1076] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/wals/813af822f7214169aed018a62884703e/wal-000000002 (ops 9-15)
I20260921 18:46:19.807294 10878 log.cc:1076] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/wals/813af822f7214169aed018a62884703e/wal-000000003 (ops 16-22)
I20260921 18:46:19.807334 10857 maintenance_manager.cc:419] P 2af2bc0080434cfeadd826373d56ccc5: Scheduling LogGCOp(813af822f7214169aed018a62884703e): free 2761400 bytes of WAL
I20260921 18:46:19.807621 10747 log_reader.cc:386] T 813af822f7214169aed018a62884703e: removed 3 log segments from log reader
I20260921 18:46:19.807729 10747 log.cc:1076] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/813af822f7214169aed018a62884703e/wal-000000001 (ops 1-9)
I20260921 18:46:19.807864 10747 log.cc:1076] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/813af822f7214169aed018a62884703e/wal-000000002 (ops 10-16)
I20260921 18:46:19.807965 10747 log.cc:1076] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/813af822f7214169aed018a62884703e/wal-000000003 (ops 17-23)
I20260921 18:46:19.808364 10878 maintenance_manager.cc:643] P fee1d4fe94924b07a24fc6a7acfd8365: LogGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20260921 18:46:19.808657 10988 maintenance_manager.cc:419] P fee1d4fe94924b07a24fc6a7acfd8365: Scheduling UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e): 570 bytes on disk
I20260921 18:46:19.809068 10747 maintenance_manager.cc:643] P 2af2bc0080434cfeadd826373d56ccc5: LogGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {}
I20260921 18:46:19.809374 10857 maintenance_manager.cc:419] P 2af2bc0080434cfeadd826373d56ccc5: Scheduling UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e): 570 bytes on disk
I20260921 18:46:19.809374 10878 maintenance_manager.cc:643] P fee1d4fe94924b07a24fc6a7acfd8365: UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20260921 18:46:19.810000 10747 maintenance_manager.cc:643] P 2af2bc0080434cfeadd826373d56ccc5: UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20260921 18:46:19.812845 3218 raft_consensus-itest-base.cc:279] Log GC complete on 2af2bc0080434cfeadd826373d56ccc5
W20260921 18:46:21.293021 10744 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.3.36.129:42703 for RPC RequestConsensusVote timed out after 1.819s (ON_OUTBOUND_QUEUE)
W20260921 18:46:21.648137 10875 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.3.36.129:42703 for RPC RequestConsensusVote timed out after 2.179s (ON_OUTBOUND_QUEUE)
I20260921 18:46:21.814064 3218 raft_consensus-itest-base.cc:295] Servers converged with original term 2
I20260921 18:46:21.814138 3218 raft_consensus-itest-base.cc:300] Resuming e11ca314edf44f8c84e10e4f0d82ce90
I20260921 18:46:21.816479 10810 raft_consensus.cc:1226] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 LEADER]: Rejecting Update request from peer e11ca314edf44f8c84e10e4f0d82ce90 for earlier term 1. Current term is 2. Ops: []
I20260921 18:46:21.817852 11015 consensus_queue.cc:1046] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 41, Time since last communication: 0.000s
I20260921 18:46:21.818007 11015 consensus_queue.cc:1225] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [LEADER]: Peer 2af2bc0080434cfeadd826373d56ccc5 log is divergent from this leader: its last log entry 2.41 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 41
I20260921 18:46:21.818073 11015 raft_consensus.cc:3032] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:21.818102 11015 raft_consensus.cc:729] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e11ca314edf44f8c84e10e4f0d82ce90, State: Running, Role: LEADER
I20260921 18:46:21.818236 11015 consensus_queue.cc:260] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } }
I20260921 18:46:21.818483 11015 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:21.820590 10940 raft_consensus.cc:1226] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Rejecting Update request from peer e11ca314edf44f8c84e10e4f0d82ce90 for earlier term 1. Current term is 2. Ops: []
I20260921 18:46:21.821666 10678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "2af2bc0080434cfeadd826373d56ccc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" is_pre_election: true
I20260921 18:46:21.821874 10675 raft_consensus.cc:1261] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Refusing update from remote peer 2af2bc0080434cfeadd826373d56ccc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260921 18:46:21.822141 10672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "2af2bc0080434cfeadd826373d56ccc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e11ca314edf44f8c84e10e4f0d82ce90"
I20260921 18:46:21.822623 11067 consensus_queue.cc:1035] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260921 18:46:21.822965 11067 consensus_queue.cc:740] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: The logs necessary to catch up peer e11ca314edf44f8c84e10e4f0d82ce90 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..34: Segment 1 which contained index 2 has been GCed)
I20260921 18:46:21.823125 10672 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af2bc0080434cfeadd826373d56ccc5 in term 2.
I20260921 18:46:21.823262 11067 raft_consensus.cc:998] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Attempting to remove follower e11ca314edf44f8c84e10e4f0d82ce90 from the Raft config. Reason: The logs necessary to catch up peer e11ca314edf44f8c84e10e4f0d82ce90 have been garbage collected. The replica will never be able to catch up
I20260921 18:46:21.823905 11067 consensus_queue.cc:237] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 2.41, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } }
I20260921 18:46:21.824934 10940 raft_consensus.cc:1261] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Refusing update from remote peer 2af2bc0080434cfeadd826373d56ccc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 41. Preceding OpId from leader: term: 2 index: 42. (index mismatch)
I20260921 18:46:21.825373 11072 consensus_queue.cc:1035] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20260921 18:46:21.826169 10672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" is_pre_election: true
I20260921 18:46:21.826328 10672 raft_consensus.cc:2379] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fee1d4fe94924b07a24fc6a7acfd8365 in current term 2: Already voted for candidate 2af2bc0080434cfeadd826373d56ccc5 in this term.
I20260921 18:46:21.826558 10875 leader_election.cc:302] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fee1d4fe94924b07a24fc6a7acfd8365; no voters: 2af2bc0080434cfeadd826373d56ccc5, e11ca314edf44f8c84e10e4f0d82ce90
I20260921 18:46:21.826912 11075 raft_consensus.cc:2726] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260921 18:46:21.827354 11067 raft_consensus.cc:2932] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 LEADER]: Committing config change with OpId 2.42: config changed from index -1 to 42, VOTER e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } }
I20260921 18:46:21.827754 10940 raft_consensus.cc:2932] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Committing config change with OpId 2.42: config changed from index -1 to 42, VOTER e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) evicted. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } }
I20260921 18:46:21.829873 10535 catalog_manager.cc:5495] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 reported cstate change: config changed from index -1 to 42, VOTER e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) evicted. New cstate: current_term: 2 leader_uuid: "2af2bc0080434cfeadd826373d56ccc5" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } }
I20260921 18:46:21.832846 10659 tablet_service.cc:1514] Processing DeleteTablet for tablet 813af822f7214169aed018a62884703e with delete_type TABLET_DATA_TOMBSTONED (TS e11ca314edf44f8c84e10e4f0d82ce90 not found in new config with opid_index 42) from {username='slave'} at 127.0.0.1:40290
I20260921 18:46:21.833242 11080 tablet_replica.cc:330] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: stopping tablet replica
I20260921 18:46:21.833575 11080 raft_consensus.cc:2229] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:21.833741 11080 raft_consensus.cc:2258] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:21.834506 11080 ts_tablet_manager.cc:1905] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:21.836261 11080 ts_tablet_manager.cc:1918] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:21.836334 11080 log.cc:1196] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/813af822f7214169aed018a62884703e
I20260921 18:46:21.836905 10522 catalog_manager.cc:4841] TS e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703): tablet 813af822f7214169aed018a62884703e (table TestTable [id=ed2d50e88d6e44ed9459c7307be5c1e1]) successfully deleted
I20260921 18:46:21.837891 10719 generic_service.cc:175] {username='slave'} at 127.0.0.1:40302 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20260921 18:46:21.838881 10810 consensus_queue.cc:237] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 2.42, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:21.840126 10940 raft_consensus.cc:1261] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Refusing update from remote peer 2af2bc0080434cfeadd826373d56ccc5: Log matching property violated. Preceding OpId in replica: term: 2 index: 42. Preceding OpId from leader: term: 2 index: 43. (index mismatch)
I20260921 18:46:21.840497 11072 consensus_queue.cc:1035] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
W20260921 18:46:21.840692 10744 consensus_peers.cc:487] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 -> Peer e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703): Couldn't send request to peer e11ca314edf44f8c84e10e4f0d82ce90. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260921 18:46:21.841902 11067 raft_consensus.cc:2932] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 LEADER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } } }
I20260921 18:46:21.842257 10940 raft_consensus.cc:2932] T 813af822f7214169aed018a62884703e P fee1d4fe94924b07a24fc6a7acfd8365 [term 2 FOLLOWER]: Committing config change with OpId 2.43: config changed from index 42 to 43, VOTER e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } } }
I20260921 18:46:21.843911 10534 catalog_manager.cc:5495] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 reported cstate change: config changed from index 42 to 43, VOTER e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129) added. New cstate: current_term: 2 leader_uuid: "2af2bc0080434cfeadd826373d56ccc5" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } } }
I20260921 18:46:22.347252 11081 ts_tablet_manager.cc:927] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Initiating tablet copy from peer 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031)
I20260921 18:46:22.347826 11081 tablet_copy_client.cc:316] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet copy: Beginning tablet copy session from remote peer at address 127.3.36.130:40031
I20260921 18:46:22.348562 10830 tablet_copy_service.cc:140] P 2af2bc0080434cfeadd826373d56ccc5: Received BeginTabletCopySession request for tablet 813af822f7214169aed018a62884703e from peer e11ca314edf44f8c84e10e4f0d82ce90 ({username='slave'} at 127.3.36.129:51927)
I20260921 18:46:22.348702 10830 tablet_copy_service.cc:161] P 2af2bc0080434cfeadd826373d56ccc5: Beginning new tablet copy session on tablet 813af822f7214169aed018a62884703e from peer e11ca314edf44f8c84e10e4f0d82ce90 at {username='slave'} at 127.3.36.129:51927: session id = e11ca314edf44f8c84e10e4f0d82ce90-813af822f7214169aed018a62884703e
I20260921 18:46:22.350217 10830 tablet_copy_source_session.cc:205] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Tablet Copy: opened 5 blocks and 3 log segments
I20260921 18:46:22.351681 11081 ts_tablet_manager.cc:1905] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260921 18:46:22.353513 11081 ts_tablet_manager.cc:1918] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260921 18:46:22.353682 11081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813af822f7214169aed018a62884703e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260921 18:46:22.354597 11081 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260921 18:46:22.354633 11081 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260921 18:46:22.354657 11081 fault_injection.cc:59] injecting fault for kudu (pid 10597): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20260921 18:46:22.395004 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 10728
I20260921 18:46:22.401366 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 10859
I20260921 18:46:22.407619 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 10506
I20260921 18:46:22.414371 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36491
--webserver_interface=127.3.36.190
--webserver_port=33289
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:36491
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20260921 18:46:22.518226 11083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:22.518450 11083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:22.518481 11083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:22.522409 11083 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:22.522461 11083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:22.522478 11083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:22.522495 11083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:22.522557 11083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:22.527110 11083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:36491
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=33289
--never_fsync=true
--heap_profile_path=/tmp/kudu.11083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:22.528085 11083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:22.529371 11083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:22.537504 11089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:22.537532 11088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:22.537622 11083 server_base.cc:1034] running on GCE node
W20260921 18:46:22.537557 11091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:22.538252 11083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:22.539314 11083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:22.540490 11083 hybrid_clock.cc:648] HybridClock initialized: now 1790016382540458 us; error 43 us; skew 500 ppm
I20260921 18:46:22.542241 11083 webserver.cc:460] Webserver started at http://127.3.36.190:33289/ using document root <none> and password file <none>
I20260921 18:46:22.542781 11083 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:22.542884 11083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:22.546355 11083 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:22.548389 11097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:22.549518 11083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:22.549698 11083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a699403814644b29a2dd1e8c4a5238b7"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:22.550176 11083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:22.561419 11083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:22.562032 11083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:22.562216 11083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:22.568436 11083 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:36491
I20260921 18:46:22.568450 11149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:36491 every 8 connection(s)
I20260921 18:46:22.569419 11083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:22.569635 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11083
I20260921 18:46:22.570163 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:40031
--local_ip_for_outbound_sockets=127.3.36.130
--tserver_master_addrs=127.3.36.190:36491
--webserver_port=34823
--webserver_interface=127.3.36.130
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260921 18:46:22.574162 11150 sys_catalog.cc:263] Verifying existing consensus state
I20260921 18:46:22.575635 11150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Bootstrap starting.
I20260921 18:46:22.588891 11150 log.cc:824] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:22.596890 11150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=5 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260921 18:46:22.597239 11150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Bootstrap complete.
I20260921 18:46:22.600994 11150 raft_consensus.cc:348] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:22.601365 11150 raft_consensus.cc:729] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a699403814644b29a2dd1e8c4a5238b7, State: Initialized, Role: FOLLOWER
I20260921 18:46:22.601889 11150 consensus_queue.cc:260] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:22.602023 11150 raft_consensus.cc:388] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:22.602108 11150 raft_consensus.cc:482] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:22.602203 11150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:22.603793 11150 raft_consensus.cc:504] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:22.604122 11150 leader_election.cc:302] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a699403814644b29a2dd1e8c4a5238b7; no voters:
I20260921 18:46:22.604475 11150 leader_election.cc:288] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260921 18:46:22.604579 11154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260921 18:46:22.604764 11154 raft_consensus.cc:686] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [term 2 LEADER]: Becoming Leader. State: Replica: a699403814644b29a2dd1e8c4a5238b7, State: Running, Role: LEADER
I20260921 18:46:22.605106 11154 consensus_queue.cc:237] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } }
I20260921 18:46:22.605666 11150 sys_catalog.cc:564] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:22.606027 11156 sys_catalog.cc:455] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a699403814644b29a2dd1e8c4a5238b7. Latest consensus state: current_term: 2 leader_uuid: "a699403814644b29a2dd1e8c4a5238b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } } }
I20260921 18:46:22.606143 11156 sys_catalog.cc:458] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:22.606366 11155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a699403814644b29a2dd1e8c4a5238b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a699403814644b29a2dd1e8c4a5238b7" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36491 } } }
I20260921 18:46:22.606472 11155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:22.606607 11158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:22.610193 11158 catalog_manager.cc:669] Loaded metadata for table TestTable [id=ed2d50e88d6e44ed9459c7307be5c1e1]
I20260921 18:46:22.611464 11158 tablet_loader.cc:96] loaded metadata for tablet 813af822f7214169aed018a62884703e (table TestTable [id=ed2d50e88d6e44ed9459c7307be5c1e1])
I20260921 18:46:22.611619 11158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:22.612221 11158 catalog_manager.cc:1259] Loaded cluster ID: 00b75e503eb24ca6b8a91ae55361fd2c
I20260921 18:46:22.612311 11158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:22.614717 11158 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:22.615386 11158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a699403814644b29a2dd1e8c4a5238b7: Loaded TSK: 0
I20260921 18:46:22.616540 11158 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260921 18:46:22.681176 11152 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260921 18:46:22.681480 11152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:22.681571 11152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:22.681655 11152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:22.685748 11152 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260921 18:46:22.685809 11152 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260921 18:46:22.685853 11152 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:22.685920 11152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:22.686017 11152 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260921 18:46:22.686069 11152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:22.690675 11152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:40031
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=34823
--tserver_master_addrs=127.3.36.190:36491
--never_fsync=true
--heap_profile_path=/tmp/kudu.11152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:22.691630 11152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:22.692893 11152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:22.701910 11176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:22.702225 11152 server_base.cc:1034] running on GCE node
W20260921 18:46:22.701886 11179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:22.701902 11177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:22.702786 11152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:22.703363 11152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:22.704552 11152 hybrid_clock.cc:648] HybridClock initialized: now 1790016382704530 us; error 40 us; skew 500 ppm
I20260921 18:46:22.706496 11152 webserver.cc:460] Webserver started at http://127.3.36.130:34823/ using document root <none> and password file <none>
I20260921 18:46:22.707190 11152 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:22.707270 11152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:22.710976 11152 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260921 18:46:22.716567 11189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:22.717720 11152 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260921 18:46:22.717833 11152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2af2bc0080434cfeadd826373d56ccc5"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:22.718318 11152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 5
Total live bytes: 136571
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:22.745627 11152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:22.746313 11152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:22.746511 11152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:22.746997 11152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:22.748136 11196 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260921 18:46:22.752144 11152 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260921 18:46:22.752200 11152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260921 18:46:22.752269 11152 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260921 18:46:22.753871 11152 ts_tablet_manager.cc:610] Registered 1 tablets
I20260921 18:46:22.753916 11152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:22.754163 11196 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Bootstrap starting.
I20260921 18:46:22.763849 11152 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:40031
I20260921 18:46:22.763906 11303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:40031 every 8 connection(s)
I20260921 18:46:22.764735 11152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:22.766010 11196 log.cc:824] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:22.766525 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11152
I20260921 18:46:22.767033 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:42703
--local_ip_for_outbound_sockets=127.3.36.129
--tserver_master_addrs=127.3.36.190:36491
--webserver_port=44629
--webserver_interface=127.3.36.129
--builtin_ntp_servers=127.3.36.148:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20260921 18:46:22.774720 11196 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 2 replicates
I20260921 18:46:22.779973 11304 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36491
I20260921 18:46:22.780184 11304 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:22.780264 11196 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=11 ignored=11} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 3 replicates
I20260921 18:46:22.780656 11304 heartbeater.cc:511] Master 127.3.36.190:36491 requested a full tablet report, sending...
I20260921 18:46:22.782856 11114 ts_manager.cc:194] Registered new tserver with Master: 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031)
I20260921 18:46:22.784420 11196 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Bootstrap replayed 3/3 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=18} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates
I20260921 18:46:22.784857 11196 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Bootstrap complete.
I20260921 18:46:22.784878 11114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:43345
I20260921 18:46:22.785902 11196 ts_tablet_manager.cc:1397] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.009s
I20260921 18:46:22.788879 11196 raft_consensus.cc:348] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:22.789443 11196 raft_consensus.cc:729] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2bc0080434cfeadd826373d56ccc5, State: Initialized, Role: FOLLOWER
I20260921 18:46:22.789944 11196 consensus_queue.cc:260] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 2.43, Last appended by leader: 43, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:22.790823 11304 heartbeater.cc:503] Master 127.3.36.190:36491 was elected leader, sending a full tablet report...
I20260921 18:46:22.790938 11196 ts_tablet_manager.cc:1428] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260921 18:46:22.866518 11305 maintenance_manager.cc:419] P 2af2bc0080434cfeadd826373d56ccc5: Scheduling UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e): 570 bytes on disk
I20260921 18:46:22.867511 11194 maintenance_manager.cc:643] P 2af2bc0080434cfeadd826373d56ccc5: UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20260921 18:46:22.887857 11309 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20260921 18:46:22.888142 11309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:22.888203 11309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:22.888255 11309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:22.892174 11309 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20260921 18:46:22.892216 11309 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20260921 18:46:22.892262 11309 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20260921 18:46:22.892311 11309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:22.892419 11309 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20260921 18:46:22.892447 11309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:22.896982 11309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:42703
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=44629
--tserver_master_addrs=127.3.36.190:36491
--never_fsync=true
--heap_profile_path=/tmp/kudu.11309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:22.897910 11309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:22.899181 11309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:22.908169 11316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:22.908164 11318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:22.908330 11309 server_base.cc:1034] running on GCE node
W20260921 18:46:22.908483 11315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:22.908888 11309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:22.909431 11309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:22.910632 11309 hybrid_clock.cc:648] HybridClock initialized: now 1790016382910602 us; error 40 us; skew 500 ppm
I20260921 18:46:22.912364 11309 webserver.cc:460] Webserver started at http://127.3.36.129:44629/ using document root <none> and password file <none>
I20260921 18:46:22.912925 11309 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:22.913039 11309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:22.916592 11309 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:46:22.918525 11324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:22.919655 11309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:22.919780 11309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e11ca314edf44f8c84e10e4f0d82ce90"
format_stamp: "Formatted at 2026-09-21 18:46:16 on dist-test-slave-l4st"
I20260921 18:46:22.920285 11309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:22.940493 11309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:22.941063 11309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:22.941253 11309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:22.941829 11309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:22.942878 11331 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260921 18:46:22.946228 11331 ts_tablet_manager.cc:1852] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260921 18:46:22.947454 11331 ts_tablet_manager.cc:1905] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:22.950801 11331 ts_tablet_manager.cc:1918] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:22.951579 11309 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260921 18:46:22.951655 11309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260921 18:46:22.951745 11309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:22.951797 11309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:22.960736 11309 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:42703
I20260921 18:46:22.960772 11438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:42703 every 8 connection(s)
I20260921 18:46:22.961534 11309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:22.963725 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11309
I20260921 18:46:22.963901 3218 test_util.cc:268] Using random seed: -1431957316
I20260921 18:46:22.971673 11439 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36491
I20260921 18:46:22.971889 11439 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:22.972370 11439 heartbeater.cc:511] Master 127.3.36.190:36491 requested a full tablet report, sending...
I20260921 18:46:22.973662 11113 ts_manager.cc:194] Registered new tserver with Master: e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
I20260921 18:46:22.974377 11113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:43411
I20260921 18:46:22.975227 3218 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260921 18:46:23.004783 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.004784 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.004783 11216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.004784 11215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.008126 11442 meta_cache.cc:302] tablet 813af822f7214169aed018a62884703e: replica fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211) has failed: Network error: Client connection negotiation failed: client connection to 127.3.36.131:44211: connect: Connection refused (error 111)
W20260921 18:46:23.021947 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.026821 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.026912 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.035869 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.045687 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.054507 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.061631 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.072806 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.082965 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.089884 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.100454 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.111781 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.125308 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.143083 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.143414 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.160501 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.173112 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.190717 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.210819 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.232908 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.256228 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.277760 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.302050 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.310420 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.338007 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.358808 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.386286 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.413304 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.440714 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.441306 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.472740 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.472805 11218 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.505110 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.528772 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.545785 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.580633 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.608938 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.644023 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.667887 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.688099 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.717275 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.752441 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.777791 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.819562 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.845057 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.887250 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.918027 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:23.959599 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
I20260921 18:46:23.976600 11439 heartbeater.cc:503] Master 127.3.36.190:36491 was elected leader, sending a full tablet report...
W20260921 18:46:23.990067 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:24.020473 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:24.035624 11442 meta_cache.cc:302] tablet 813af822f7214169aed018a62884703e: replica fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211) has failed: Network error: Client connection negotiation failed: client connection to 127.3.36.131:44211: connect: Connection refused (error 111)
W20260921 18:46:24.054991 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:24.098646 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
W20260921 18:46:24.127823 11217 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51770: Illegal state: replica 2af2bc0080434cfeadd826373d56ccc5 is not leader of this config: current role FOLLOWER
I20260921 18:46:24.131601 11455 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:46:24.131769 11455 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:24.132805 11455 leader_election.cc:288] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211), e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
W20260921 18:46:24.133457 11190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.131:44211: connect: Connection refused (error 111)
W20260921 18:46:24.134456 11190 leader_election.cc:334] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211): Network error: Client connection negotiation failed: client connection to 127.3.36.131:44211: connect: Connection refused (error 111)
I20260921 18:46:24.136941 11393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "2af2bc0080434cfeadd826373d56ccc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" is_pre_election: true
I20260921 18:46:24.138036 11393 raft_consensus.cc:1789] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260921 18:46:24.138420 11393 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2af2bc0080434cfeadd826373d56ccc5 in term 2.
I20260921 18:46:24.138826 11191 leader_election.cc:302] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2af2bc0080434cfeadd826373d56ccc5, e11ca314edf44f8c84e10e4f0d82ce90; no voters: fee1d4fe94924b07a24fc6a7acfd8365
I20260921 18:46:24.139089 11455 raft_consensus.cc:2781] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260921 18:46:24.139190 11455 raft_consensus.cc:482] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:46:24.139266 11455 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 2 FOLLOWER]: Advancing to term 3
I20260921 18:46:24.141177 11455 raft_consensus.cc:504] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:24.141517 11455 leader_election.cc:288] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 3 election: Requested vote from peers fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211), e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703)
I20260921 18:46:24.141829 11393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "813af822f7214169aed018a62884703e" candidate_uuid: "2af2bc0080434cfeadd826373d56ccc5" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "e11ca314edf44f8c84e10e4f0d82ce90"
I20260921 18:46:24.141929 11393 raft_consensus.cc:1789] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20260921 18:46:24.141995 11393 raft_consensus.cc:3037] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 2 FOLLOWER]: Advancing to term 3
W20260921 18:46:24.142482 11190 leader_election.cc:334] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211): Network error: Client connection negotiation failed: client connection to 127.3.36.131:44211: connect: Connection refused (error 111)
I20260921 18:46:24.143044 11393 raft_consensus.cc:2454] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2af2bc0080434cfeadd826373d56ccc5 in term 3.
I20260921 18:46:24.143325 11191 leader_election.cc:302] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2af2bc0080434cfeadd826373d56ccc5, e11ca314edf44f8c84e10e4f0d82ce90; no voters: fee1d4fe94924b07a24fc6a7acfd8365
I20260921 18:46:24.143537 11455 raft_consensus.cc:2781] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260921 18:46:24.143746 11455 raft_consensus.cc:686] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [term 3 LEADER]: Becoming Leader. State: Replica: 2af2bc0080434cfeadd826373d56ccc5, State: Running, Role: LEADER
I20260921 18:46:24.144059 11455 consensus_queue.cc:237] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 2.43, Last appended by leader: 43, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:24.146322 11114 catalog_manager.cc:5495] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "2af2bc0080434cfeadd826373d56ccc5" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } } }
W20260921 18:46:24.167905 11191 consensus_peers.cc:487] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 -> Peer e11ca314edf44f8c84e10e4f0d82ce90 (127.3.36.129:42703): Couldn't send request to peer e11ca314edf44f8c84e10e4f0d82ce90. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260921 18:46:24.168107 11190 consensus_peers.cc:487] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5 -> Peer fee1d4fe94924b07a24fc6a7acfd8365 (127.3.36.131:44211): Couldn't send request to peer fee1d4fe94924b07a24fc6a7acfd8365. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.131:44211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260921 18:46:24.694994 11467 tablet_replica.cc:330] stopping tablet replica
I20260921 18:46:24.695216 11467 raft_consensus.cc:2229] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:24.695353 11467 raft_consensus.cc:2258] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:24.695581 11467 ts_tablet_manager.cc:927] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Initiating tablet copy from peer 2af2bc0080434cfeadd826373d56ccc5 (127.3.36.130:40031)
I20260921 18:46:24.696115 11467 tablet_copy_client.cc:316] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet copy: Beginning tablet copy session from remote peer at address 127.3.36.130:40031
I20260921 18:46:24.702275 11278 tablet_copy_service.cc:140] P 2af2bc0080434cfeadd826373d56ccc5: Received BeginTabletCopySession request for tablet 813af822f7214169aed018a62884703e from peer e11ca314edf44f8c84e10e4f0d82ce90 ({username='slave'} at 127.3.36.129:59827)
I20260921 18:46:24.702458 11278 tablet_copy_service.cc:161] P 2af2bc0080434cfeadd826373d56ccc5: Beginning new tablet copy session on tablet 813af822f7214169aed018a62884703e from peer e11ca314edf44f8c84e10e4f0d82ce90 at {username='slave'} at 127.3.36.129:59827: session id = e11ca314edf44f8c84e10e4f0d82ce90-813af822f7214169aed018a62884703e
I20260921 18:46:24.704589 11278 tablet_copy_source_session.cc:205] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Tablet Copy: opened 5 blocks and 3 log segments
I20260921 18:46:24.707062 11467 ts_tablet_manager.cc:1905] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260921 18:46:24.709205 11467 ts_tablet_manager.cc:1918] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260921 18:46:24.709689 11467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 813af822f7214169aed018a62884703e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:24.711539 11467 tablet_copy_client.cc:791] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet copy: Starting download of 5 data blocks...
I20260921 18:46:24.719848 11467 tablet_copy_client.cc:655] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet copy: Starting download of 3 WAL segments...
I20260921 18:46:24.728322 11467 tablet_copy_client.cc:531] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260921 18:46:24.730265 11467 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Bootstrap starting.
I20260921 18:46:24.738808 11467 log.cc:824] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:24.745209 11467 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20260921 18:46:24.749207 11467 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=11 ignored=11} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20260921 18:46:24.753350 11467 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Bootstrap replayed 3/3 log segments. Stats: ops{read=25 overwritten=0 applied=20 ignored=18} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20260921 18:46:24.753808 11467 tablet_bootstrap.cc:492] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Bootstrap complete.
I20260921 18:46:24.754572 11467 ts_tablet_manager.cc:1397] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20260921 18:46:24.756134 11467 raft_consensus.cc:348] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:24.757833 11467 raft_consensus.cc:729] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11ca314edf44f8c84e10e4f0d82ce90, State: Initialized, Role: FOLLOWER
I20260921 18:46:24.758330 11467 consensus_queue.cc:260] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 3.48, Last appended by leader: 48, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "2af2bc0080434cfeadd826373d56ccc5" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 40031 } } peers { permanent_uuid: "fee1d4fe94924b07a24fc6a7acfd8365" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 44211 } } peers { permanent_uuid: "e11ca314edf44f8c84e10e4f0d82ce90" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 42703 } attrs { } }
I20260921 18:46:24.759215 11467 ts_tablet_manager.cc:1428] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:46:24.759950 11277 tablet_copy_service.cc:342] P 2af2bc0080434cfeadd826373d56ccc5: Request end of tablet copy session e11ca314edf44f8c84e10e4f0d82ce90-813af822f7214169aed018a62884703e received from {username='slave'} at 127.3.36.129:59827
I20260921 18:46:24.760089 11277 tablet_copy_service.cc:433] P 2af2bc0080434cfeadd826373d56ccc5: ending tablet copy session e11ca314edf44f8c84e10e4f0d82ce90-813af822f7214169aed018a62884703e on tablet 813af822f7214169aed018a62884703e with peer e11ca314edf44f8c84e10e4f0d82ce90
I20260921 18:46:24.765916 11440 maintenance_manager.cc:419] P e11ca314edf44f8c84e10e4f0d82ce90: Scheduling UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e): 570 bytes on disk
I20260921 18:46:24.766722 11329 maintenance_manager.cc:643] P e11ca314edf44f8c84e10e4f0d82ce90: UndoDeltaBlockGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20260921 18:46:25.265957 11393 raft_consensus.cc:1203] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 2.43->[3.44-3.48] Dedup: 3.48->[]
I20260921 18:46:25.267884 11481 mvcc.cc:205] Tried to move back new op lower bound from 7331907109545091072 to 7331907109456121856. Current Snapshot: MvccSnapshot[applied={T|T < 7331907109545091072}]
I20260921 18:46:25.269248 11483 mvcc.cc:205] Tried to move back new op lower bound from 7331907110486147072 to 7331907109456121856. Current Snapshot: MvccSnapshot[applied={T|T < 7331907109545091072}]
I20260921 18:46:25.274765 11484 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51770 (ReqId={client: c2cb3eec157a49c282ddf31539b97d8f, seq_no=2, attempt_no=45}) took 1078 ms. Trace:
I20260921 18:46:25.274927 11485 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51770 (ReqId={client: c2cb3eec157a49c282ddf31539b97d8f, seq_no=0, attempt_no=46}) took 1005 ms. Trace:
I20260921 18:46:25.274992 11482 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51770 (ReqId={client: c2cb3eec157a49c282ddf31539b97d8f, seq_no=3, attempt_no=44}) took 1111 ms. Trace:
I20260921 18:46:25.275085 11484 rpcz_store.cc:276] 0921 18:46:24.196542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:24.196614 (+ 72us) service_pool.cc:227] Handling call
0921 18:46:25.274752 (+1078138us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:24.196976 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 813af822f7214169aed018a62884703e
0921 18:46:24.197073 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:24.197076 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:46:24.197078 (+ 2us) tablet.cc:660] Decoding operations
0921 18:46:24.197273 (+ 195us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:24.197300 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:24.197302 (+ 2us) tablet.cc:683] Acquiring locks for 50 operations
0921 18:46:24.197454 (+ 152us) tablet.cc:699] Row locks acquired
0921 18:46:24.197455 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:46:24.197543 (+ 88us) write_op.cc:270] Start()
0921 18:46:24.197570 (+ 27us) write_op.cc:276] Timestamp: P: 1790016384197469 usec, L: 0
0921 18:46:24.197572 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:46:24.197715 (+ 143us) log.cc:842] Serialized 2455 byte log entry
0921 18:46:25.267926 (+1070211us) op_driver.cc:444] REPLICATION: finished
0921 18:46:25.268307 (+ 381us) write_op.cc:301] APPLY: starting
0921 18:46:25.268322 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:25.269599 (+ 1277us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:25.269617 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:25.274362 (+ 4745us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:25.274398 (+ 36us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=50,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:25.274402 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:46:25.274493 (+ 91us) log.cc:842] Serialized 319 byte log entry
0921 18:46:25.274677 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:25.274696 (+ 19us) write_op.cc:454] Released schema lock
0921 18:46:25.274702 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":157,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":6,"mutex_wait_us":32,"num_ops":50,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":855,"prepare.run_wall_time_us":856,"raft.queue_time_us":113,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":1070340,"spinlock_wait_cycles":404736,"thread_start_us":80,"threads_started":1}]]}
I20260921 18:46:25.275198 11485 rpcz_store.cc:276] 0921 18:46:24.269122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:24.269199 (+ 77us) service_pool.cc:227] Handling call
0921 18:46:25.274914 (+1005715us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:24.269619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 813af822f7214169aed018a62884703e
0921 18:46:24.269701 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:24.269705 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:46:24.269706 (+ 1us) tablet.cc:660] Decoding operations
0921 18:46:24.269903 (+ 197us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:24.269909 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:24.269911 (+ 2us) tablet.cc:683] Acquiring locks for 50 operations
0921 18:46:24.270071 (+ 160us) tablet.cc:699] Row locks acquired
0921 18:46:24.270072 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:46:24.270123 (+ 51us) write_op.cc:270] Start()
0921 18:46:24.270158 (+ 35us) write_op.cc:276] Timestamp: P: 1790016384270095 usec, L: 0
0921 18:46:24.270159 (+ 1us) op_driver.cc:331] REPLICATION: starting
0921 18:46:24.270296 (+ 137us) log.cc:842] Serialized 2455 byte log entry
0921 18:46:25.268248 (+997952us) op_driver.cc:444] REPLICATION: finished
0921 18:46:25.268435 (+ 187us) write_op.cc:301] APPLY: starting
0921 18:46:25.268445 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:25.270742 (+ 2297us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:25.270754 (+ 12us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:25.274649 (+ 3895us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:25.274697 (+ 48us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=49,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:25.274701 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:46:25.274794 (+ 93us) log.cc:842] Serialized 319 byte log entry
0921 18:46:25.274842 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:25.274861 (+ 19us) write_op.cc:454] Released schema lock
0921 18:46:25.274868 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":150,"cfile_cache_hit":2,"cfile_cache_hit_bytes":4127,"mutex_wait_us":56,"num_ops":50,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":822,"raft.queue_time_us":188,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":998076,"spinlock_wait_cycles":1280,"thread_start_us":68,"threads_started":1}]]}
I20260921 18:46:25.275542 11482 rpcz_store.cc:276] 0921 18:46:24.163902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:24.163966 (+ 64us) service_pool.cc:227] Handling call
0921 18:46:25.274983 (+1111017us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:24.164515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 813af822f7214169aed018a62884703e
0921 18:46:24.164624 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:24.164631 (+ 7us) write_op.cc:435] Acquired schema lock
0921 18:46:24.164653 (+ 22us) tablet.cc:660] Decoding operations
0921 18:46:24.165349 (+ 696us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:24.165399 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:24.165405 (+ 6us) tablet.cc:683] Acquiring locks for 50 operations
0921 18:46:24.166290 (+ 885us) tablet.cc:699] Row locks acquired
0921 18:46:24.166292 (+ 2us) write_op.cc:260] PREPARE: finished
0921 18:46:24.166346 (+ 54us) write_op.cc:270] Start()
0921 18:46:24.166378 (+ 32us) write_op.cc:276] Timestamp: P: 1790016384166282 usec, L: 0
0921 18:46:24.166380 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:46:24.166549 (+ 169us) log.cc:842] Serialized 2455 byte log entry
0921 18:46:25.267785 (+1101236us) op_driver.cc:444] REPLICATION: finished
0921 18:46:25.268219 (+ 434us) write_op.cc:301] APPLY: starting
0921 18:46:25.268242 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:25.271144 (+ 2902us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:25.271172 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:25.274757 (+ 3585us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:25.274795 (+ 38us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=49,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:25.274799 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:46:25.274874 (+ 75us) log.cc:842] Serialized 319 byte log entry
0921 18:46:25.274897 (+ 23us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:25.274934 (+ 37us) write_op.cc:454] Released schema lock
0921 18:46:25.274947 (+ 13us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":368,"cfile_cache_hit":2,"cfile_cache_hit_bytes":4127,"mutex_wait_us":71,"num_ops":50,"prepare.queue_time_us":227,"prepare.run_cpu_time_us":2195,"prepare.run_wall_time_us":2195,"raft.queue_time_us":141,"raft.run_cpu_time_us":644,"raft.run_wall_time_us":657,"replication_time_us":1101362,"spinlock_wait_cycles":381056,"thread_start_us":173,"threads_started":2}]]}
I20260921 18:46:25.870748 11440 maintenance_manager.cc:419] P e11ca314edf44f8c84e10e4f0d82ce90: Scheduling LogGCOp(813af822f7214169aed018a62884703e): free 920390 bytes of WAL
I20260921 18:46:25.871033 11329 log_reader.cc:386] T 813af822f7214169aed018a62884703e: removed 1 log segments from log reader
I20260921 18:46:25.871119 11329 log.cc:1076] T 813af822f7214169aed018a62884703e P e11ca314edf44f8c84e10e4f0d82ce90: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/813af822f7214169aed018a62884703e/wal-000000001 (ops 24-30)
I20260921 18:46:25.871498 11329 maintenance_manager.cc:643] P e11ca314edf44f8c84e10e4f0d82ce90: LogGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260921 18:46:25.878130 11305 maintenance_manager.cc:419] P 2af2bc0080434cfeadd826373d56ccc5: Scheduling LogGCOp(813af822f7214169aed018a62884703e): free 920390 bytes of WAL
I20260921 18:46:25.878376 11194 log_reader.cc:386] T 813af822f7214169aed018a62884703e: removed 1 log segments from log reader
I20260921 18:46:25.878475 11194 log.cc:1076] T 813af822f7214169aed018a62884703e P 2af2bc0080434cfeadd826373d56ccc5: Deleting log segment in path: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/813af822f7214169aed018a62884703e/wal-000000001 (ops 24-30)
I20260921 18:46:25.878893 11194 maintenance_manager.cc:643] P 2af2bc0080434cfeadd826373d56ccc5: LogGCOp(813af822f7214169aed018a62884703e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260921 18:46:26.010200 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11309
I20260921 18:46:26.016441 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11152
I20260921 18:46:26.021987 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11083
2026-09-21T18:46:26Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (9613 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20260921 18:46:26.045429 3218 test_util.cc:268] Using random seed: -1428875791
I20260921 18:46:26.046270 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:26.046314 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:26.046326 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:46:26.046337 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:46:26.046371 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:26Z Disabled control of system clock
I20260921 18:46:26.054900 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:38589
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:38589 with env {}
W20260921 18:46:26.158134 11506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:26.158418 11506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:26.158481 11506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:26.162424 11506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:26.162503 11506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:26.162540 11506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:26.162578 11506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:26.167165 11506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:38589
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:26.168221 11506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:26.169575 11506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:26.177279 11511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:26.177263 11512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:26.177443 11506 server_base.cc:1034] running on GCE node
W20260921 18:46:26.177285 11514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:26.178139 11506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:26.179122 11506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:26.180301 11506 hybrid_clock.cc:648] HybridClock initialized: now 1790016386180287 us; error 37 us; skew 500 ppm
I20260921 18:46:26.182044 11506 webserver.cc:460] Webserver started at http://127.3.36.190:43951/ using document root <none> and password file <none>
I20260921 18:46:26.182570 11506 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:26.182669 11506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:26.182884 11506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:26.184612 11506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.185143 11506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.188510 11506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260921 18:46:26.190716 11520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.191829 11506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:26.191959 11506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.192060 11506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:26.207814 11506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:26.208365 11506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:26.208551 11506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:26.215011 11506 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:38589
I20260921 18:46:26.215027 11572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:38589 every 8 connection(s)
I20260921 18:46:26.215890 11506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:26.218933 11573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:26.220148 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11506
I20260921 18:46:26.220264 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:26.224612 11573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1: Bootstrap starting.
I20260921 18:46:26.227074 11573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:26.227885 11573 log.cc:824] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:26.229662 11573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1: No bootstrap required, opened a new log
I20260921 18:46:26.233064 11573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 38589 } }
I20260921 18:46:26.233258 11573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:26.233355 11573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a953cfaf6894bbfaab4cfa6540e8fe1, State: Initialized, Role: FOLLOWER
I20260921 18:46:26.233785 11573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 38589 } }
I20260921 18:46:26.233928 11573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:26.234015 11573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:26.234129 11573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:26.235934 11573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 38589 } }
I20260921 18:46:26.236274 11573 leader_election.cc:302] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a953cfaf6894bbfaab4cfa6540e8fe1; no voters:
I20260921 18:46:26.236577 11573 leader_election.cc:288] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:26.236709 11578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:26.236918 11578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [term 1 LEADER]: Becoming Leader. State: Replica: 4a953cfaf6894bbfaab4cfa6540e8fe1, State: Running, Role: LEADER
I20260921 18:46:26.237268 11578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 38589 } }
I20260921 18:46:26.237804 11573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:26.238765 11580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a953cfaf6894bbfaab4cfa6540e8fe1. Latest consensus state: current_term: 1 leader_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 38589 } } }
I20260921 18:46:26.239013 11579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a953cfaf6894bbfaab4cfa6540e8fe1" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 38589 } } }
I20260921 18:46:26.239122 11580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:26.239225 11579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:26.240005 11587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:26.242532 11587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:26.247313 11587 catalog_manager.cc:1347] Generated new cluster ID: 64ae42fdaeba4b3fab3e174a6d434a84
I20260921 18:46:26.247372 11587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:26.253697 11587 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:26.254498 11587 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:26.259557 11587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a953cfaf6894bbfaab4cfa6540e8fe1: Generated new TSK 0
I20260921 18:46:26.260174 11587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:26.263216 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:38589
--builtin_ntp_servers=127.3.36.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20260921 18:46:26.371268 11597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:26.371495 11597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:26.371529 11597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:26.375790 11597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:26.375876 11597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:26.380529 11597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:38589
--never_fsync=true
--heap_profile_path=/tmp/kudu.11597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:26.381536 11597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:26.382864 11597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:26.391788 11603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:26.391961 11597 server_base.cc:1034] running on GCE node
W20260921 18:46:26.391786 11602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:26.391836 11605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:26.392534 11597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:26.393143 11597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:26.394341 11597 hybrid_clock.cc:648] HybridClock initialized: now 1790016386394297 us; error 53 us; skew 500 ppm
I20260921 18:46:26.396189 11597 webserver.cc:460] Webserver started at http://127.3.36.129:32991/ using document root <none> and password file <none>
I20260921 18:46:26.396734 11597 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:26.396824 11597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:26.397045 11597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:26.398901 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f962d53cb35d469d8a76ba78f550bf81"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.399449 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f962d53cb35d469d8a76ba78f550bf81"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.402864 11597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260921 18:46:26.405148 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.406343 11597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:26.406468 11597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f962d53cb35d469d8a76ba78f550bf81"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.406575 11597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:26.418078 11597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:26.418771 11597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:26.419027 11597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:26.419597 11597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:26.420509 11597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:26.420590 11597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.420662 11597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:26.420699 11597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.429149 11597 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:34645
I20260921 18:46:26.429221 11724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:34645 every 8 connection(s)
I20260921 18:46:26.430109 11597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:26.438905 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11597
I20260921 18:46:26.439033 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:26.440207 11725 heartbeater.cc:347] Connected to a master server at 127.3.36.190:38589
I20260921 18:46:26.440433 11725 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:26.441039 11725 heartbeater.cc:511] Master 127.3.36.190:38589 requested a full tablet report, sending...
I20260921 18:46:26.441431 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:38589
--builtin_ntp_servers=127.3.36.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20260921 18:46:26.442685 11536 ts_manager.cc:194] Registered new tserver with Master: f962d53cb35d469d8a76ba78f550bf81 (127.3.36.129:34645)
I20260921 18:46:26.444188 11536 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:55993
W20260921 18:46:26.548986 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:26.549216 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:26.549249 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:26.553215 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:26.553296 11728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:26.557803 11728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:38589
--never_fsync=true
--heap_profile_path=/tmp/kudu.11728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:26.558687 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:26.559892 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:26.568630 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:26.568630 11734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:26.568656 11733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:26.569012 11728 server_base.cc:1034] running on GCE node
I20260921 18:46:26.569429 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:26.570060 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:26.571256 11728 hybrid_clock.cc:648] HybridClock initialized: now 1790016386571227 us; error 51 us; skew 500 ppm
I20260921 18:46:26.573069 11728 webserver.cc:460] Webserver started at http://127.3.36.130:39055/ using document root <none> and password file <none>
I20260921 18:46:26.573693 11728 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:26.573781 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:26.574028 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:26.575759 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "12296e7c22ba4c88a008d207ddbf57de"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.576328 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "12296e7c22ba4c88a008d207ddbf57de"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.579802 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:26.581990 11742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.583148 11728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:26.583274 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "12296e7c22ba4c88a008d207ddbf57de"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.583374 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:26.597728 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:26.598311 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:26.598465 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:26.598990 11728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:26.599829 11728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:26.599896 11728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.599962 11728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:26.599997 11728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.607872 11728 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:46825
I20260921 18:46:26.607893 11855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:46825 every 8 connection(s)
I20260921 18:46:26.608724 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:26.618419 11856 heartbeater.cc:347] Connected to a master server at 127.3.36.190:38589
I20260921 18:46:26.618631 11856 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:26.618721 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11728
I20260921 18:46:26.618848 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:26.619189 11856 heartbeater.cc:511] Master 127.3.36.190:38589 requested a full tablet report, sending...
I20260921 18:46:26.620101 11537 ts_manager.cc:194] Registered new tserver with Master: 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825)
I20260921 18:46:26.620723 11537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:53785
I20260921 18:46:26.621060 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:38589
--builtin_ntp_servers=127.3.36.148:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20260921 18:46:26.726442 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:26.726673 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:26.726708 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:26.730896 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:26.731011 11859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:26.735446 11859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:38589
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:26.736382 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:26.737731 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:26.746714 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:26.746724 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:26.746749 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:26.747167 11859 server_base.cc:1034] running on GCE node
I20260921 18:46:26.747543 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:26.748119 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:26.749302 11859 hybrid_clock.cc:648] HybridClock initialized: now 1790016386749282 us; error 30 us; skew 500 ppm
I20260921 18:46:26.751010 11859 webserver.cc:460] Webserver started at http://127.3.36.131:40049/ using document root <none> and password file <none>
I20260921 18:46:26.751555 11859 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:26.751643 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:26.751848 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:26.753676 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e1e6a7c5f970486594e6cabb61ac7cd6"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.754204 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e1e6a7c5f970486594e6cabb61ac7cd6"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.757557 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:26.759752 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.760875 11859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:26.761006 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e1e6a7c5f970486594e6cabb61ac7cd6"
format_stamp: "Formatted at 2026-09-21 18:46:26 on dist-test-slave-l4st"
I20260921 18:46:26.761108 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:26.772006 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:26.772576 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:26.772748 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:26.773242 11859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:26.774135 11859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:26.774209 11859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.774263 11859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:26.774299 11859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:26.782750 11986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:45877 every 8 connection(s)
I20260921 18:46:26.782730 11859 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:45877
I20260921 18:46:26.783613 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:26.786612 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 11859
I20260921 18:46:26.786728 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:26.793406 11987 heartbeater.cc:347] Connected to a master server at 127.3.36.190:38589
I20260921 18:46:26.793655 11987 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:26.794127 11987 heartbeater.cc:511] Master 127.3.36.190:38589 requested a full tablet report, sending...
I20260921 18:46:26.795046 11537 ts_manager.cc:194] Registered new tserver with Master: e1e6a7c5f970486594e6cabb61ac7cd6 (127.3.36.131:45877)
I20260921 18:46:26.795624 11537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:52861
I20260921 18:46:26.800727 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:26.809037 11537 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38880:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260921 18:46:26.812232 11537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:46:26.828795 11790 tablet_service.cc:1467] Processing CreateTablet for tablet 520ac84dbe46421baee523fede9c7a1d (DEFAULT_TABLE table=TestTable [id=9242d8b4f2144893a8f3e9b12ef2b7a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:26.828642 11659 tablet_service.cc:1467] Processing CreateTablet for tablet 520ac84dbe46421baee523fede9c7a1d (DEFAULT_TABLE table=TestTable [id=9242d8b4f2144893a8f3e9b12ef2b7a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:26.829766 11659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520ac84dbe46421baee523fede9c7a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:26.829766 11790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520ac84dbe46421baee523fede9c7a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:26.831260 11921 tablet_service.cc:1467] Processing CreateTablet for tablet 520ac84dbe46421baee523fede9c7a1d (DEFAULT_TABLE table=TestTable [id=9242d8b4f2144893a8f3e9b12ef2b7a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260921 18:46:26.832036 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520ac84dbe46421baee523fede9c7a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:26.836308 12005 tablet_bootstrap.cc:492] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Bootstrap starting.
I20260921 18:46:26.836920 12004 tablet_bootstrap.cc:492] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: Bootstrap starting.
I20260921 18:46:26.837178 12006 tablet_bootstrap.cc:492] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Bootstrap starting.
I20260921 18:46:26.839107 12006 tablet_bootstrap.cc:654] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:26.839780 12006 log.cc:824] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:26.840013 12004 tablet_bootstrap.cc:654] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:26.840852 12005 tablet_bootstrap.cc:654] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:26.841634 12006 tablet_bootstrap.cc:492] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: No bootstrap required, opened a new log
I20260921 18:46:26.841634 12005 log.cc:824] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:26.841813 12006 ts_tablet_manager.cc:1397] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:26.842270 12004 log.cc:824] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:26.843541 12005 tablet_bootstrap.cc:492] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: No bootstrap required, opened a new log
I20260921 18:46:26.843708 12005 ts_tablet_manager.cc:1397] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20260921 18:46:26.843720 12004 tablet_bootstrap.cc:492] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: No bootstrap required, opened a new log
I20260921 18:46:26.843916 12004 ts_tablet_manager.cc:1397] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260921 18:46:26.846191 12006 raft_consensus.cc:348] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.846441 12006 raft_consensus.cc:374] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:26.846509 12006 raft_consensus.cc:729] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e6a7c5f970486594e6cabb61ac7cd6, State: Initialized, Role: FOLLOWER
I20260921 18:46:26.847055 12006 consensus_queue.cc:260] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.847759 11987 heartbeater.cc:503] Master 127.3.36.190:38589 was elected leader, sending a full tablet report...
I20260921 18:46:26.848058 12006 ts_tablet_manager.cc:1428] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:46:26.847873 12005 raft_consensus.cc:348] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.848142 12005 raft_consensus.cc:374] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:26.848188 12005 raft_consensus.cc:729] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f962d53cb35d469d8a76ba78f550bf81, State: Initialized, Role: FOLLOWER
I20260921 18:46:26.848096 12004 raft_consensus.cc:348] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.848327 12004 raft_consensus.cc:374] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:26.848382 12004 raft_consensus.cc:729] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12296e7c22ba4c88a008d207ddbf57de, State: Initialized, Role: FOLLOWER
I20260921 18:46:26.848598 12005 consensus_queue.cc:260] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.848748 12004 consensus_queue.cc:260] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.849232 11725 heartbeater.cc:503] Master 127.3.36.190:38589 was elected leader, sending a full tablet report...
I20260921 18:46:26.849488 12005 ts_tablet_manager.cc:1428] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260921 18:46:26.850028 11856 heartbeater.cc:503] Master 127.3.36.190:38589 was elected leader, sending a full tablet report...
I20260921 18:46:26.850418 12004 ts_tablet_manager.cc:1428] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260921 18:46:26.859575 11857 tablet.cc:2369] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:26.890964 12011 raft_consensus.cc:482] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:46:26.891173 12011 raft_consensus.cc:504] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.892227 12011 leader_election.cc:288] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1e6a7c5f970486594e6cabb61ac7cd6 (127.3.36.131:45877), 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825)
I20260921 18:46:26.897246 11810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "f962d53cb35d469d8a76ba78f550bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12296e7c22ba4c88a008d207ddbf57de" is_pre_election: true
I20260921 18:46:26.897339 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "f962d53cb35d469d8a76ba78f550bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: true
I20260921 18:46:26.897362 12010 raft_consensus.cc:482] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260921 18:46:26.897588 11810 raft_consensus.cc:2454] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f962d53cb35d469d8a76ba78f550bf81 in term 0.
I20260921 18:46:26.897555 12010 raft_consensus.cc:504] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.897960 11941 raft_consensus.cc:2454] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f962d53cb35d469d8a76ba78f550bf81 in term 0.
I20260921 18:46:26.898048 11615 leader_election.cc:302] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12296e7c22ba4c88a008d207ddbf57de, f962d53cb35d469d8a76ba78f550bf81; no voters:
I20260921 18:46:26.898322 12011 raft_consensus.cc:2781] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260921 18:46:26.898397 12011 raft_consensus.cc:482] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260921 18:46:26.898438 12011 raft_consensus.cc:3037] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:26.898634 12010 leader_election.cc:288] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f962d53cb35d469d8a76ba78f550bf81 (127.3.36.129:34645), 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825)
I20260921 18:46:26.899631 12011 raft_consensus.cc:504] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.900010 12011 leader_election.cc:288] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [CANDIDATE]: Term 1 election: Requested vote from peers e1e6a7c5f970486594e6cabb61ac7cd6 (127.3.36.131:45877), 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825)
I20260921 18:46:26.900365 11810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "f962d53cb35d469d8a76ba78f550bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12296e7c22ba4c88a008d207ddbf57de"
I20260921 18:46:26.900473 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "f962d53cb35d469d8a76ba78f550bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6"
I20260921 18:46:26.900476 11810 raft_consensus.cc:3037] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:26.900563 11941 raft_consensus.cc:3037] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:26.901628 11810 raft_consensus.cc:2454] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f962d53cb35d469d8a76ba78f550bf81 in term 1.
I20260921 18:46:26.901628 11941 raft_consensus.cc:2454] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f962d53cb35d469d8a76ba78f550bf81 in term 1.
I20260921 18:46:26.902043 11615 leader_election.cc:302] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1e6a7c5f970486594e6cabb61ac7cd6, f962d53cb35d469d8a76ba78f550bf81; no voters:
I20260921 18:46:26.902323 12011 raft_consensus.cc:2781] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:26.902559 12011 raft_consensus.cc:686] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [term 1 LEADER]: Becoming Leader. State: Replica: f962d53cb35d469d8a76ba78f550bf81, State: Running, Role: LEADER
I20260921 18:46:26.903151 12011 consensus_queue.cc:237] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } }
I20260921 18:46:26.903944 11810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12296e7c22ba4c88a008d207ddbf57de" is_pre_election: true
I20260921 18:46:26.904112 11810 raft_consensus.cc:2379] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1e6a7c5f970486594e6cabb61ac7cd6 in current term 1: Already voted for candidate f962d53cb35d469d8a76ba78f550bf81 in this term.
I20260921 18:46:26.904263 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: true
I20260921 18:46:26.905027 11877 leader_election.cc:302] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1e6a7c5f970486594e6cabb61ac7cd6; no voters: 12296e7c22ba4c88a008d207ddbf57de, f962d53cb35d469d8a76ba78f550bf81
I20260921 18:46:26.905283 12010 raft_consensus.cc:2726] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260921 18:46:26.906528 11536 catalog_manager.cc:5495] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 reported cstate change: term changed from 0 to 1, leader changed from <none> to f962d53cb35d469d8a76ba78f550bf81 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "f962d53cb35d469d8a76ba78f550bf81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f962d53cb35d469d8a76ba78f550bf81" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 34645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:26.916115 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:26.917436 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f962d53cb35d469d8a76ba78f550bf81 to finish bootstrapping
I20260921 18:46:26.922878 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12296e7c22ba4c88a008d207ddbf57de to finish bootstrapping
I20260921 18:46:26.927644 3218 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e1e6a7c5f970486594e6cabb61ac7cd6 to finish bootstrapping
W20260921 18:46:26.931236 11726 tablet.cc:2369] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:26.932905 11719 generic_service.cc:175] {username='slave'} at 127.0.0.1:37286 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260921 18:46:26.933825 11850 generic_service.cc:175] {username='slave'} at 127.0.0.1:39268 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260921 18:46:26.934660 11981 generic_service.cc:175] {username='slave'} at 127.0.0.1:36526 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20260921 18:46:26.944496 11810 raft_consensus.cc:1261] T 520ac84dbe46421baee523fede9c7a1d P 12296e7c22ba4c88a008d207ddbf57de [term 1 FOLLOWER]: Refusing update from remote peer f962d53cb35d469d8a76ba78f550bf81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:26.944561 11941 raft_consensus.cc:1261] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6 [term 1 FOLLOWER]: Refusing update from remote peer f962d53cb35d469d8a76ba78f550bf81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260921 18:46:26.945076 12011 consensus_queue.cc:1035] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12296e7c22ba4c88a008d207ddbf57de" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:26.945271 12015 consensus_queue.cc:1035] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260921 18:46:26.950809 12019 mvcc.cc:205] Tried to move back new op lower bound from 7331907120919216128 to 7331907120758427648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907120919216128}]
I20260921 18:46:26.951778 12021 mvcc.cc:205] Tried to move back new op lower bound from 7331907120919216128 to 7331907120758427648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907120919216128}]
I20260921 18:46:26.953015 12020 mvcc.cc:205] Tried to move back new op lower bound from 7331907120919216128 to 7331907120758427648. Current Snapshot: MvccSnapshot[applied={T|T < 7331907120919216128}]
I20260921 18:46:26.963165 11981 generic_service.cc:175] {username='slave'} at 127.0.0.1:36526 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20260921 18:46:26.963804 11981 generic_service.cc:175] {username='slave'} at 127.0.0.1:36526 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260921 18:46:26.964413 11981 generic_service.cc:175] {username='slave'} at 127.0.0.1:36526 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260921 18:46:26.965132 11719 generic_service.cc:175] {username='slave'} at 127.0.0.1:37286 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20260921 18:46:26.965860 11719 generic_service.cc:175] {username='slave'} at 127.0.0.1:37286 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20260921 18:46:26.966501 11719 generic_service.cc:175] {username='slave'} at 127.0.0.1:37286 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20260921 18:46:26.966780 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11728
I20260921 18:46:26.975721 3218 test_util.cc:268] Using random seed: -1427945497
I20260921 18:46:26.983527 3218 test_workload.cc:401] TestWorkload: Skipping table creation because table TestTable already exists
W20260921 18:46:26.991637 12016 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260921 18:46:26.992074 11615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111)
W20260921 18:46:26.992939 11615 consensus_peers.cc:487] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 -> Peer 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825): Couldn't send request to peer 12296e7c22ba4c88a008d207ddbf57de. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260921 18:46:26.993224 12022 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260921 18:46:27.034684 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.036430 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.088054 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.089352 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.140682 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.141870 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.193255 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.194377 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.245718 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.246891 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.298251 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.299360 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.350613 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.351631 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.402822 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.403900 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.455144 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.456272 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.507826 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.508980 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.560438 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.561586 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.612979 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.614264 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.665688 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.666867 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.718215 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.719362 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.770726 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.771884 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.823169 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.824311 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.875720 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.876857 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.928309 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.929538 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:27.980904 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:27.982071 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:27.996877 11615 consensus_peers.cc:487] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 -> Peer 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825): Couldn't send request to peer 12296e7c22ba4c88a008d207ddbf57de. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260921 18:46:28.033540 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.034715 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.086061 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.087424 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.138893 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.140199 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.191430 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.192570 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.193040 12036 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37298 (ReqId={client: 1da853e87ca041138ba581a4b75327c5, seq_no=0, attempt_no=0}) took 1203 ms. Trace:
I20260921 18:46:28.193127 12036 rpcz_store.cc:276] 0921 18:46:26.989808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:26.989863 (+ 55us) service_pool.cc:227] Handling call
0921 18:46:28.193027 (+1203164us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:26.990847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 520ac84dbe46421baee523fede9c7a1d
0921 18:46:26.990912 (+ 65us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:26.990915 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:46:26.990917 (+ 2us) tablet.cc:660] Decoding operations
0921 18:46:26.990975 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:26.990982 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:26.990984 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:46:26.991015 (+ 31us) tablet.cc:699] Row locks acquired
0921 18:46:26.991016 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:46:26.991120 (+ 104us) write_op.cc:270] Start()
0921 18:46:26.991141 (+ 21us) write_op.cc:276] Timestamp: P: 1790016386991093 usec, L: 0
0921 18:46:26.991142 (+ 1us) op_driver.cc:331] REPLICATION: starting
0921 18:46:26.991317 (+ 175us) log.cc:842] Serialized 345 byte log entry
0921 18:46:28.192010 (+1200693us) op_driver.cc:444] REPLICATION: finished
0921 18:46:28.192238 (+ 228us) write_op.cc:301] APPLY: starting
0921 18:46:28.192257 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:28.192291 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:28.192305 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:28.192385 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:28.192414 (+ 29us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:28.192418 (+ 4us) write_op.cc:312] APPLY: finished
0921 18:46:28.192485 (+ 67us) log.cc:842] Serialized 22 byte log entry
0921 18:46:28.192714 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:28.192727 (+ 13us) write_op.cc:454] Released schema lock
0921 18:46:28.192733 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":172,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":584,"raft.queue_time_us":120,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":310,"replication_time_us":1200844,"spinlock_wait_cycles":9472,"thread_start_us":105,"threads_started":1}]]}
I20260921 18:46:28.244086 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.245205 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.296650 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.297930 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.349268 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.350553 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:28.396924 12016 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260921 18:46:28.398138 12022 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260921 18:46:28.401849 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.402873 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.454134 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.455299 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.506632 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.507834 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.559228 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.560369 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.611706 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.612850 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.664373 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.665658 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.716949 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.718058 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.769394 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.770607 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.822052 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.823230 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.874816 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.876013 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:28.927484 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.928748 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:28.946794 11615 consensus_peers.cc:487] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 -> Peer 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825): Couldn't send request to peer 12296e7c22ba4c88a008d207ddbf57de. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260921 18:46:28.980168 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:28.981318 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.032846 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.034021 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.085511 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.086702 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.138276 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.139460 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.190685 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.191781 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.243131 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.244361 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.295790 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.296955 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.348338 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.349432 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.400767 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.401891 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.453233 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.454373 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.505653 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.506762 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.558171 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.559448 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.598037 12042 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37298 (ReqId={client: 1da853e87ca041138ba581a4b75327c5, seq_no=1, attempt_no=0}) took 1202 ms. Trace:
I20260921 18:46:29.598132 12042 rpcz_store.cc:276] 0921 18:46:28.395353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:28.395439 (+ 86us) service_pool.cc:227] Handling call
0921 18:46:29.598004 (+1202565us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:28.396211 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 520ac84dbe46421baee523fede9c7a1d
0921 18:46:28.396306 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:28.396309 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:46:28.396310 (+ 1us) tablet.cc:660] Decoding operations
0921 18:46:28.396372 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:28.396380 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:28.396382 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:46:28.396413 (+ 31us) tablet.cc:699] Row locks acquired
0921 18:46:28.396414 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:46:28.396446 (+ 32us) write_op.cc:270] Start()
0921 18:46:28.396468 (+ 22us) write_op.cc:276] Timestamp: P: 1790016388396381 usec, L: 0
0921 18:46:28.396469 (+ 1us) op_driver.cc:331] REPLICATION: starting
0921 18:46:28.396666 (+ 197us) log.cc:842] Serialized 354 byte log entry
0921 18:46:29.597350 (+1200684us) op_driver.cc:444] REPLICATION: finished
0921 18:46:29.597615 (+ 265us) write_op.cc:301] APPLY: starting
0921 18:46:29.597631 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:29.597657 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:29.597671 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:29.597725 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:29.597748 (+ 23us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:29.597750 (+ 2us) write_op.cc:312] APPLY: finished
0921 18:46:29.597811 (+ 61us) log.cc:842] Serialized 22 byte log entry
0921 18:46:29.597907 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:29.597921 (+ 14us) write_op.cc:454] Released schema lock
0921 18:46:29.597926 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":207,"num_ops":1,"prepare.queue_time_us":185,"prepare.run_cpu_time_us":564,"prepare.run_wall_time_us":602,"raft.queue_time_us":109,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":180,"replication_time_us":1200855,"thread_start_us":205,"threads_started":2}]]}
I20260921 18:46:29.610683 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.611723 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.663311 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.664670 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.716094 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.717308 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.768775 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.769955 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:29.801265 12016 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260921 18:46:29.802541 12022 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260921 18:46:29.821354 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.822536 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.873934 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.875187 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.926582 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.927752 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:29.979120 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:29.980273 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:30.002759 11615 consensus_peers.cc:487] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 -> Peer 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825): Couldn't send request to peer 12296e7c22ba4c88a008d207ddbf57de. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260921 18:46:30.031608 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.032778 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.084395 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.085734 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.137271 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.138523 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.189739 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.190873 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.242197 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.243366 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.294801 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.296180 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.347559 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.348757 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.400118 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.401383 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.452723 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.453876 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.505259 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.506377 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.557834 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.558964 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.610211 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.611416 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.662696 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.663766 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.715220 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.716390 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.767614 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.768739 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.820129 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.821237 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.872530 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.873641 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:30.924984 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.926167 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:30.945727 11615 consensus_peers.cc:487] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 -> Peer 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825): Couldn't send request to peer 12296e7c22ba4c88a008d207ddbf57de. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260921 18:46:30.977504 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:30.978806 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.002547 12048 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37298 (ReqId={client: 1da853e87ca041138ba581a4b75327c5, seq_no=2, attempt_no=0}) took 1202 ms. Trace:
I20260921 18:46:31.002653 12048 rpcz_store.cc:276] 0921 18:46:29.799843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:29.799938 (+ 95us) service_pool.cc:227] Handling call
0921 18:46:31.002526 (+1202588us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:29.800532 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 520ac84dbe46421baee523fede9c7a1d
0921 18:46:29.800613 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:29.800617 (+ 4us) write_op.cc:435] Acquired schema lock
0921 18:46:29.800618 (+ 1us) tablet.cc:660] Decoding operations
0921 18:46:29.800696 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:29.800703 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:29.800705 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:46:29.800737 (+ 32us) tablet.cc:699] Row locks acquired
0921 18:46:29.800738 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:46:29.800768 (+ 30us) write_op.cc:270] Start()
0921 18:46:29.800790 (+ 22us) write_op.cc:276] Timestamp: P: 1790016389800712 usec, L: 0
0921 18:46:29.800792 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:46:29.800946 (+ 154us) log.cc:842] Serialized 354 byte log entry
0921 18:46:31.001773 (+1200827us) op_driver.cc:444] REPLICATION: finished
0921 18:46:31.002121 (+ 348us) write_op.cc:301] APPLY: starting
0921 18:46:31.002141 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:31.002179 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:31.002194 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:31.002250 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:31.002274 (+ 24us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:31.002277 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:46:31.002338 (+ 61us) log.cc:842] Serialized 22 byte log entry
0921 18:46:31.002423 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:31.002438 (+ 15us) write_op.cc:454] Released schema lock
0921 18:46:31.002443 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":278,"num_ops":1,"prepare.queue_time_us":193,"prepare.run_cpu_time_us":557,"prepare.run_wall_time_us":557,"raft.queue_time_us":148,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":214,"replication_time_us":1200955,"spinlock_wait_cycles":14592,"thread_start_us":160,"threads_started":2}]]}
I20260921 18:46:31.030292 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.031425 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.082937 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.084115 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.135617 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.136755 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.188009 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.189311 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:31.205233 12016 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20260921 18:46:31.206354 12022 log.cc:508] T 520ac84dbe46421baee523fede9c7a1d P e1e6a7c5f970486594e6cabb61ac7cd6: Injecting 600ms of latency in SegmentAllocator::Sync()
I20260921 18:46:31.240564 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.241595 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.293006 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.294178 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.345711 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.346920 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.398171 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.399211 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.450413 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.451504 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.502959 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.504490 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.555770 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.557032 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.608494 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.609619 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.660967 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.662055 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.713402 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.714529 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.765980 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.767326 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.818789 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.819847 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.871161 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.872251 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:31.914541 11615 consensus_peers.cc:487] T 520ac84dbe46421baee523fede9c7a1d P f962d53cb35d469d8a76ba78f550bf81 -> Peer 12296e7c22ba4c88a008d207ddbf57de (127.3.36.130:46825): Couldn't send request to peer 12296e7c22ba4c88a008d207ddbf57de. Status: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260921 18:46:31.923560 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.924662 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:31.975872 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:31.976948 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:32.028374 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:32.029628 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:32.080884 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:32.081986 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
W20260921 18:46:32.103686 11615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.36.130:46825: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20260921 18:46:32.133325 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:32.134449 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:32.185976 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:32.187127 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:32.238480 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "f962d53cb35d469d8a76ba78f550bf81" is_pre_election: false
I20260921 18:46:32.239696 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520ac84dbe46421baee523fede9c7a1d" candidate_uuid: "12296e7c22ba4c88a008d207ddbf57de" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "e1e6a7c5f970486594e6cabb61ac7cd6" is_pre_election: false
I20260921 18:46:32.406441 12054 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37298 (ReqId={client: 1da853e87ca041138ba581a4b75327c5, seq_no=3, attempt_no=0}) took 1202 ms. Trace:
I20260921 18:46:32.406541 12054 rpcz_store.cc:276] 0921 18:46:31.204073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0921 18:46:31.204136 (+ 63us) service_pool.cc:227] Handling call
0921 18:46:32.406414 (+1202278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0921 18:46:31.204628 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 520ac84dbe46421baee523fede9c7a1d
0921 18:46:31.204711 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0921 18:46:31.204714 (+ 3us) write_op.cc:435] Acquired schema lock
0921 18:46:31.204715 (+ 1us) tablet.cc:660] Decoding operations
0921 18:46:31.204783 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0921 18:46:31.204791 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0921 18:46:31.204792 (+ 1us) tablet.cc:683] Acquiring locks for 1 operations
0921 18:46:31.204821 (+ 29us) tablet.cc:699] Row locks acquired
0921 18:46:31.204822 (+ 1us) write_op.cc:260] PREPARE: finished
0921 18:46:31.204849 (+ 27us) write_op.cc:270] Start()
0921 18:46:31.204887 (+ 38us) write_op.cc:276] Timestamp: P: 1790016391204809 usec, L: 0
0921 18:46:31.204889 (+ 2us) op_driver.cc:331] REPLICATION: starting
0921 18:46:31.205011 (+ 122us) log.cc:842] Serialized 354 byte log entry
0921 18:46:32.405666 (+1200655us) op_driver.cc:444] REPLICATION: finished
0921 18:46:32.405898 (+ 232us) write_op.cc:301] APPLY: starting
0921 18:46:32.405914 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0921 18:46:32.405940 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0921 18:46:32.405976 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0921 18:46:32.406105 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
0921 18:46:32.406130 (+ 25us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0921 18:46:32.406133 (+ 3us) write_op.cc:312] APPLY: finished
0921 18:46:32.406223 (+ 90us) log.cc:842] Serialized 22 byte log entry
0921 18:46:32.406308 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0921 18:46:32.406326 (+ 18us) write_op.cc:454] Released schema lock
0921 18:46:32.406331 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":178,"num_ops":1,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":503,"prepare.run_wall_time_us":504,"raft.queue_time_us":126,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":183,"replication_time_us":1200748,"thread_start_us":166,"threads_started":2}]]}
I20260921 18:46:32.608211 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11597
I20260921 18:46:32.614094 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11859
I20260921 18:46:32.619522 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 11506
2026-09-21T18:46:32Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (6594 ms)
[----------] 12 tests from RaftConsensusElectionITest (164862 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20260921 18:46:32.640463 3218 test_util.cc:268] Using random seed: -1422280758
I20260921 18:46:32.641227 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:32.641300 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:32.641320 3218 ts_itest-base.cc:114] 1 tablet servers
I20260921 18:46:32.641345 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:46:32.641369 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:32Z Disabled control of system clock
I20260921 18:46:32.649945 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36569
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:36569
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260921 18:46:32.757524 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:32.757838 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:32.757911 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:32.762017 12065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:32.762107 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:32.762143 12065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:32.762208 12065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:32.766885 12065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:36569
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:36569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:32.768054 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:32.769498 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:32.778153 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:32.778182 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:32.778247 12065 server_base.cc:1034] running on GCE node
W20260921 18:46:32.778194 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:32.779018 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:32.780094 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:32.781261 12065 hybrid_clock.cc:648] HybridClock initialized: now 1790016392781239 us; error 35 us; skew 500 ppm
I20260921 18:46:32.786433 12065 webserver.cc:460] Webserver started at http://127.3.36.190:35157/ using document root <none> and password file <none>
I20260921 18:46:32.787039 12065 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:32.787102 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:32.787344 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:32.789278 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c3b2411d74764ab29a03c8eb69a7928d"
format_stamp: "Formatted at 2026-09-21 18:46:32 on dist-test-slave-l4st"
I20260921 18:46:32.789891 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c3b2411d74764ab29a03c8eb69a7928d"
format_stamp: "Formatted at 2026-09-21 18:46:32 on dist-test-slave-l4st"
I20260921 18:46:32.793597 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260921 18:46:32.795900 12079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:32.797091 12065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:32.797225 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c3b2411d74764ab29a03c8eb69a7928d"
format_stamp: "Formatted at 2026-09-21 18:46:32 on dist-test-slave-l4st"
I20260921 18:46:32.797328 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:32.821525 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:32.822202 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:32.822397 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:32.828995 12065 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:36569
I20260921 18:46:32.829005 12131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:36569 every 8 connection(s)
I20260921 18:46:32.829936 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:32.832955 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:32.835428 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 12065
I20260921 18:46:32.835543 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:32.838498 12132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d: Bootstrap starting.
I20260921 18:46:32.840595 12132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:32.841339 12132 log.cc:824] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:32.843256 12132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d: No bootstrap required, opened a new log
I20260921 18:46:32.847235 12132 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b2411d74764ab29a03c8eb69a7928d" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36569 } }
I20260921 18:46:32.847427 12132 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:32.847460 12132 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b2411d74764ab29a03c8eb69a7928d, State: Initialized, Role: FOLLOWER
I20260921 18:46:32.847787 12132 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b2411d74764ab29a03c8eb69a7928d" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36569 } }
I20260921 18:46:32.847910 12132 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:32.847968 12132 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:32.848037 12132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:32.848716 12132 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b2411d74764ab29a03c8eb69a7928d" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36569 } }
I20260921 18:46:32.849080 12132 leader_election.cc:302] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3b2411d74764ab29a03c8eb69a7928d; no voters:
I20260921 18:46:32.849396 12132 leader_election.cc:288] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:32.849541 12137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:32.849754 12137 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [term 1 LEADER]: Becoming Leader. State: Replica: c3b2411d74764ab29a03c8eb69a7928d, State: Running, Role: LEADER
I20260921 18:46:32.850108 12137 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b2411d74764ab29a03c8eb69a7928d" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36569 } }
I20260921 18:46:32.850613 12132 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:32.851660 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3b2411d74764ab29a03c8eb69a7928d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b2411d74764ab29a03c8eb69a7928d" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36569 } } }
I20260921 18:46:32.851779 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:32.852561 12139 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b2411d74764ab29a03c8eb69a7928d. Latest consensus state: current_term: 1 leader_uuid: "c3b2411d74764ab29a03c8eb69a7928d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3b2411d74764ab29a03c8eb69a7928d" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 36569 } } }
I20260921 18:46:32.852675 12139 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d [sys.catalog]: This master's current role is: LEADER
W20260921 18:46:32.855011 12153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260921 18:46:32.855098 12153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260921 18:46:32.855176 12146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:32.855818 12146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:32.860601 12146 catalog_manager.cc:1347] Generated new cluster ID: 57c60d76be844cd2a1b766b519822919
I20260921 18:46:32.860659 12146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:32.885529 12146 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:32.886727 12146 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:32.893868 12146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3b2411d74764ab29a03c8eb69a7928d: Generated new TSK 0
I20260921 18:46:32.894506 12146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:32.905555 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36569
--builtin_ntp_servers=127.3.36.148:43483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260921 18:46:33.012908 12156 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:33.013165 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:33.013186 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:33.013217 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:33.017171 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:33.017251 12156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:33.021847 12156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:43483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:36569
--never_fsync=true
--heap_profile_path=/tmp/kudu.12156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:33.022876 12156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:33.024188 12156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:33.033200 12162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:33.033174 12164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:33.033179 12161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:33.033525 12156 server_base.cc:1034] running on GCE node
I20260921 18:46:33.033946 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:33.034546 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:33.035727 12156 hybrid_clock.cc:648] HybridClock initialized: now 1790016393035716 us; error 32 us; skew 500 ppm
I20260921 18:46:33.037361 12156 webserver.cc:460] Webserver started at http://127.3.36.129:43033/ using document root <none> and password file <none>
I20260921 18:46:33.037962 12156 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:33.038053 12156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:33.038275 12156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:33.040052 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.040630 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.044159 12156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260921 18:46:33.046327 12170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.047439 12156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:33.047569 12156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.047678 12156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:33.063956 12156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:33.064570 12156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:33.064760 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:33.065292 12156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:33.066200 12156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:33.066269 12156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.066339 12156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:33.066385 12156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.074919 12156 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:33933
I20260921 18:46:33.074949 12283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:33933 every 8 connection(s)
I20260921 18:46:33.075796 12156 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:33.081701 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 12156
I20260921 18:46:33.081832 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:33.086071 12284 heartbeater.cc:347] Connected to a master server at 127.3.36.190:36569
I20260921 18:46:33.086306 12284 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:33.086812 12284 heartbeater.cc:511] Master 127.3.36.190:36569 requested a full tablet report, sending...
I20260921 18:46:33.088164 12096 ts_manager.cc:194] Registered new tserver with Master: 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129:33933)
I20260921 18:46:33.089238 12096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:57853
I20260921 18:46:33.095587 3218 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260921 18:46:33.104285 3218 external_mini_cluster.cc:930] 1 TS(s) registered with all masters
I20260921 18:46:33.104357 3218 test_util.cc:268] Using random seed: -1421816860
I20260921 18:46:33.114104 12096 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50394:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260921 18:46:33.132395 12217 tablet_service.cc:1467] Processing CreateTablet for tablet 023aeebf56ac4343ae85ee0cd0143142 (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260921 18:46:33.132304 12211 tablet_service.cc:1467] Processing CreateTablet for tablet 9317b5ff865f44119cd49a14f6d3c0df (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260921 18:46:33.132859 12214 tablet_service.cc:1467] Processing CreateTablet for tablet 300ec264dcaa4e629d3e5b58cb9c075c (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260921 18:46:33.132304 12218 tablet_service.cc:1467] Processing CreateTablet for tablet 2799a472698141d9b47a10dd23ba117b (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260921 18:46:33.132304 12213 tablet_service.cc:1467] Processing CreateTablet for tablet 89f21fc8e17f41a3ad0a74155b16c342 (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260921 18:46:33.132304 12216 tablet_service.cc:1467] Processing CreateTablet for tablet f143987f328f4e90a3cff621952cbc56 (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260921 18:46:33.132402 12212 tablet_service.cc:1467] Processing CreateTablet for tablet 6f255063787043b29c0fe8fd77da4670 (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260921 18:46:33.132805 12209 tablet_service.cc:1467] Processing CreateTablet for tablet 9e255f8b0bea4c2483bf5ebd83fa15ad (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260921 18:46:33.132484 12210 tablet_service.cc:1467] Processing CreateTablet for tablet 237b470309044f92b99c3f6a109f4132 (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260921 18:46:33.133896 12217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023aeebf56ac4343ae85ee0cd0143142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.132395 12215 tablet_service.cc:1467] Processing CreateTablet for tablet fb6ec88552c643ed8da84c69f522dc56 (DEFAULT_TABLE table=test-workload [id=9c75ff715bc9447187941882ee84834e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260921 18:46:33.134267 12215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6ec88552c643ed8da84c69f522dc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.134418 12212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f255063787043b29c0fe8fd77da4670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.135104 12214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300ec264dcaa4e629d3e5b58cb9c075c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.135331 12209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e255f8b0bea4c2483bf5ebd83fa15ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.135478 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2799a472698141d9b47a10dd23ba117b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.135738 12210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 237b470309044f92b99c3f6a109f4132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.136023 12216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f143987f328f4e90a3cff621952cbc56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.136193 12213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f21fc8e17f41a3ad0a74155b16c342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.136518 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9317b5ff865f44119cd49a14f6d3c0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.144732 12299 tablet_bootstrap.cc:492] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.146581 12299 tablet_bootstrap.cc:654] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.147558 12299 log.cc:824] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:33.149061 12299 tablet_bootstrap.cc:492] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.149263 12299 ts_tablet_manager.cc:1397] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260921 18:46:33.152313 12299 raft_consensus.cc:348] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.152508 12299 raft_consensus.cc:729] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.152791 12299 consensus_queue.cc:260] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.153355 12284 heartbeater.cc:503] Master 127.3.36.190:36569 was elected leader, sending a full tablet report...
I20260921 18:46:33.153705 12299 ts_tablet_manager.cc:1428] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260921 18:46:33.153913 12299 tablet_bootstrap.cc:492] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.154774 12299 tablet_bootstrap.cc:654] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.155920 12299 tablet_bootstrap.cc:492] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.156023 12299 ts_tablet_manager.cc:1397] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:33.156646 12299 raft_consensus.cc:348] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.156739 12299 raft_consensus.cc:729] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.156831 12299 consensus_queue.cc:260] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.157119 12299 ts_tablet_manager.cc:1428] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:33.157280 12299 tablet_bootstrap.cc:492] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.158197 12299 tablet_bootstrap.cc:654] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.159512 12299 tablet_bootstrap.cc:492] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.159586 12299 ts_tablet_manager.cc:1397] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:33.159976 12299 raft_consensus.cc:348] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.160074 12299 raft_consensus.cc:729] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.160184 12299 consensus_queue.cc:260] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.160432 12299 ts_tablet_manager.cc:1428] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.160605 12299 tablet_bootstrap.cc:492] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.161408 12299 tablet_bootstrap.cc:654] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.162762 12299 tablet_bootstrap.cc:492] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.162856 12299 ts_tablet_manager.cc:1397] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:33.163298 12299 raft_consensus.cc:348] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.163400 12299 raft_consensus.cc:729] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.163486 12299 consensus_queue.cc:260] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.163708 12299 ts_tablet_manager.cc:1428] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.163866 12299 tablet_bootstrap.cc:492] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.164608 12299 tablet_bootstrap.cc:654] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.165736 12299 tablet_bootstrap.cc:492] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.165813 12299 ts_tablet_manager.cc:1397] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:33.166136 12299 raft_consensus.cc:348] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.166213 12299 raft_consensus.cc:729] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.166292 12299 consensus_queue.cc:260] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.166515 12299 ts_tablet_manager.cc:1428] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.166672 12299 tablet_bootstrap.cc:492] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.167455 12299 tablet_bootstrap.cc:654] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.168569 12299 tablet_bootstrap.cc:492] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.168648 12299 ts_tablet_manager.cc:1397] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:33.169198 12299 raft_consensus.cc:348] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.169270 12299 raft_consensus.cc:729] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.169353 12299 consensus_queue.cc:260] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.169617 12299 ts_tablet_manager.cc:1428] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:33.169795 12299 tablet_bootstrap.cc:492] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.170666 12299 tablet_bootstrap.cc:654] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.171592 12299 tablet_bootstrap.cc:492] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.171685 12299 ts_tablet_manager.cc:1397] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:33.172024 12299 raft_consensus.cc:348] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.172119 12299 raft_consensus.cc:729] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.172216 12299 consensus_queue.cc:260] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.172478 12299 ts_tablet_manager.cc:1428] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.172633 12299 tablet_bootstrap.cc:492] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.173460 12299 tablet_bootstrap.cc:654] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.174504 12299 tablet_bootstrap.cc:492] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.174584 12299 ts_tablet_manager.cc:1397] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:33.174988 12299 raft_consensus.cc:348] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.175087 12299 raft_consensus.cc:729] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.175187 12299 consensus_queue.cc:260] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.175420 12299 ts_tablet_manager.cc:1428] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.175582 12299 tablet_bootstrap.cc:492] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.176352 12299 tablet_bootstrap.cc:654] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.177367 12299 tablet_bootstrap.cc:492] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.177446 12299 ts_tablet_manager.cc:1397] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:33.177944 12299 raft_consensus.cc:348] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.178050 12299 raft_consensus.cc:729] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.178144 12299 consensus_queue.cc:260] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.178390 12299 ts_tablet_manager.cc:1428] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.178587 12299 tablet_bootstrap.cc:492] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: Bootstrap starting.
I20260921 18:46:33.179298 12299 tablet_bootstrap.cc:654] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.180593 12299 tablet_bootstrap.cc:492] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: No bootstrap required, opened a new log
I20260921 18:46:33.180675 12299 ts_tablet_manager.cc:1397] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:33.181030 12299 raft_consensus.cc:348] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.181123 12299 raft_consensus.cc:729] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.181223 12299 consensus_queue.cc:260] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.181474 12299 ts_tablet_manager.cc:1428] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:33.218729 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9317b5ff865f44119cd49a14f6d3c0df"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.218911 12238 raft_consensus.cc:482] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.218997 12238 raft_consensus.cc:3037] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.219887 12238 raft_consensus.cc:504] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.220194 12238 leader_election.cc:302] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.220364 12238 leader_election.cc:288] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.220443 12301 raft_consensus.cc:2781] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.220728 12301 raft_consensus.cc:686] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.221076 12301 consensus_queue.cc:237] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.224023 12095 catalog_manager.cc:5495] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.228813 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89f21fc8e17f41a3ad0a74155b16c342"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.228940 12238 raft_consensus.cc:482] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.229018 12238 raft_consensus.cc:3037] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.229883 12238 raft_consensus.cc:504] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.230022 12238 leader_election.cc:302] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.230135 12238 leader_election.cc:288] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.230170 12301 raft_consensus.cc:2781] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.230237 12301 raft_consensus.cc:686] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.230351 12301 consensus_queue.cc:237] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.231655 12095 catalog_manager.cc:5495] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.236693 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2799a472698141d9b47a10dd23ba117b"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.236785 12238 raft_consensus.cc:482] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.236878 12238 raft_consensus.cc:3037] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.237684 12238 raft_consensus.cc:504] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.237807 12238 leader_election.cc:302] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.237914 12238 leader_election.cc:288] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.237982 12301 raft_consensus.cc:2781] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.238075 12301 raft_consensus.cc:686] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.238181 12301 consensus_queue.cc:237] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.239724 12095 catalog_manager.cc:5495] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.245333 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e255f8b0bea4c2483bf5ebd83fa15ad"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.245432 12238 raft_consensus.cc:482] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.245584 12238 raft_consensus.cc:3037] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.246336 12238 raft_consensus.cc:504] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.246434 12238 leader_election.cc:302] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.246521 12238 leader_election.cc:288] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.246627 12301 raft_consensus.cc:2781] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.246829 12301 raft_consensus.cc:686] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.246982 12301 consensus_queue.cc:237] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.248262 12095 catalog_manager.cc:5495] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.253589 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f255063787043b29c0fe8fd77da4670"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.253679 12238 raft_consensus.cc:482] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.253712 12238 raft_consensus.cc:3037] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.254336 12238 raft_consensus.cc:504] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.254429 12238 leader_election.cc:302] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.254539 12238 leader_election.cc:288] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.254573 12301 raft_consensus.cc:2781] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.254662 12301 raft_consensus.cc:686] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.254773 12301 consensus_queue.cc:237] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.255939 12095 catalog_manager.cc:5495] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.261793 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f143987f328f4e90a3cff621952cbc56"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.261924 12238 raft_consensus.cc:482] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.262001 12238 raft_consensus.cc:3037] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.262770 12238 raft_consensus.cc:504] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.262916 12238 leader_election.cc:302] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.263065 12238 leader_election.cc:288] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.263126 12301 raft_consensus.cc:2781] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.263257 12301 raft_consensus.cc:686] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.263377 12301 consensus_queue.cc:237] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.264734 12095 catalog_manager.cc:5495] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.269860 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "237b470309044f92b99c3f6a109f4132"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.269963 12238 raft_consensus.cc:482] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.270000 12238 raft_consensus.cc:3037] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.270721 12238 raft_consensus.cc:504] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.270834 12238 leader_election.cc:302] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.270931 12238 leader_election.cc:288] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.270969 12301 raft_consensus.cc:2781] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.271042 12301 raft_consensus.cc:686] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.271160 12301 consensus_queue.cc:237] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.272528 12095 catalog_manager.cc:5495] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.277405 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb6ec88552c643ed8da84c69f522dc56"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.277556 12238 raft_consensus.cc:482] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.277634 12238 raft_consensus.cc:3037] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.278318 12238 raft_consensus.cc:504] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.278455 12238 leader_election.cc:302] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.278563 12238 leader_election.cc:288] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.278623 12301 raft_consensus.cc:2781] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.278730 12301 raft_consensus.cc:686] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.278847 12301 consensus_queue.cc:237] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.280069 12095 catalog_manager.cc:5495] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.285074 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "023aeebf56ac4343ae85ee0cd0143142"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.285223 12238 raft_consensus.cc:482] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.285300 12238 raft_consensus.cc:3037] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.286183 12238 raft_consensus.cc:504] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.286319 12238 leader_election.cc:302] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.286434 12238 leader_election.cc:288] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.286496 12301 raft_consensus.cc:2781] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.286603 12301 raft_consensus.cc:686] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.286736 12301 consensus_queue.cc:237] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.288012 12095 catalog_manager.cc:5495] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.293104 12238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "300ec264dcaa4e629d3e5b58cb9c075c"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.293192 12238 raft_consensus.cc:482] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:33.293226 12238 raft_consensus.cc:3037] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.293938 12238 raft_consensus.cc:504] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.294034 12238 leader_election.cc:302] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f21d7f54eb423ebddce8f5fdac8e41; no voters:
I20260921 18:46:33.294102 12238 leader_election.cc:288] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.294167 12301 raft_consensus.cc:2781] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.294317 12301 raft_consensus.cc:686] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Leader. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.294495 12301 consensus_queue.cc:237] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.295869 12095 catalog_manager.cc:5495] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76f21d7f54eb423ebddce8f5fdac8e41 (127.3.36.129). New cstate: current_term: 1 leader_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260921 18:46:33.301324 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 9317b5ff865f44119cd49a14f6d3c0df with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.301771 12325 tablet_replica.cc:330] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.302114 12325 raft_consensus.cc:2229] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:33.302326 12325 raft_consensus.cc:2258] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.303506 12325 ts_tablet_manager.cc:1905] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.305542 12325 ts_tablet_manager.cc:1918] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.305655 12325 log.cc:1196] T 9317b5ff865f44119cd49a14f6d3c0df P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/9317b5ff865f44119cd49a14f6d3c0df
I20260921 18:46:33.311215 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f21fc8e17f41a3ad0a74155b16c342 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.311375 12325 tablet_replica.cc:330] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.311503 12325 raft_consensus.cc:2229] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:33.311614 12325 raft_consensus.cc:2258] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.312150 12325 ts_tablet_manager.cc:1905] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.313644 12325 ts_tablet_manager.cc:1918] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.313730 12325 log.cc:1196] T 89f21fc8e17f41a3ad0a74155b16c342 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/89f21fc8e17f41a3ad0a74155b16c342
I20260921 18:46:33.319253 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 2799a472698141d9b47a10dd23ba117b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.319424 12325 tablet_replica.cc:330] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.319559 12325 raft_consensus.cc:2229] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:33.319670 12325 raft_consensus.cc:2258] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.320206 12325 ts_tablet_manager.cc:1905] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.321889 12325 ts_tablet_manager.cc:1918] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.321974 12325 log.cc:1196] T 2799a472698141d9b47a10dd23ba117b P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/2799a472698141d9b47a10dd23ba117b
I20260921 18:46:33.327600 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e255f8b0bea4c2483bf5ebd83fa15ad with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.327772 12325 tablet_replica.cc:330] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.327903 12325 raft_consensus.cc:2229] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:33.328053 12325 raft_consensus.cc:2258] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.328680 12325 ts_tablet_manager.cc:1905] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.330412 12325 ts_tablet_manager.cc:1918] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.330503 12325 log.cc:1196] T 9e255f8b0bea4c2483bf5ebd83fa15ad P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e255f8b0bea4c2483bf5ebd83fa15ad
I20260921 18:46:33.336333 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f255063787043b29c0fe8fd77da4670 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.336542 12325 tablet_replica.cc:330] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.336683 12325 raft_consensus.cc:2229] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:33.336820 12325 raft_consensus.cc:2258] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.337572 12325 ts_tablet_manager.cc:1905] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.339287 12325 ts_tablet_manager.cc:1918] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.339382 12325 log.cc:1196] T 6f255063787043b29c0fe8fd77da4670 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f255063787043b29c0fe8fd77da4670
I20260921 18:46:33.345463 12238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f143987f328f4e90a3cff621952cbc56"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.345745 12238 raft_consensus.cc:580] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Received request to step down
I20260921 18:46:33.345805 12238 raft_consensus.cc:3032] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:33.345865 12238 raft_consensus.cc:729] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.346004 12238 consensus_queue.cc:260] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.346095 12238 raft_consensus.cc:3037] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:33.352128 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet f143987f328f4e90a3cff621952cbc56 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.352380 12325 tablet_replica.cc:330] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.352537 12325 raft_consensus.cc:2229] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:33.352681 12325 raft_consensus.cc:2258] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.353351 12325 ts_tablet_manager.cc:1905] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.355149 12325 ts_tablet_manager.cc:1918] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.355249 12325 log.cc:1196] T f143987f328f4e90a3cff621952cbc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/f143987f328f4e90a3cff621952cbc56
I20260921 18:46:33.361289 12238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "237b470309044f92b99c3f6a109f4132"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.361452 12238 raft_consensus.cc:580] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Received request to step down
I20260921 18:46:33.361563 12238 raft_consensus.cc:3032] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:33.361609 12238 raft_consensus.cc:729] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.361711 12238 consensus_queue.cc:260] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.361785 12238 raft_consensus.cc:3037] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:33.367386 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 237b470309044f92b99c3f6a109f4132 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.367635 12325 tablet_replica.cc:330] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.367808 12325 raft_consensus.cc:2229] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:33.367949 12325 raft_consensus.cc:2258] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.368641 12325 ts_tablet_manager.cc:1905] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.370574 12325 ts_tablet_manager.cc:1918] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.370637 12325 log.cc:1196] T 237b470309044f92b99c3f6a109f4132 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/237b470309044f92b99c3f6a109f4132
I20260921 18:46:33.376446 12238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb6ec88552c643ed8da84c69f522dc56"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.376562 12238 raft_consensus.cc:580] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Received request to step down
I20260921 18:46:33.376644 12238 raft_consensus.cc:3032] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:33.376698 12238 raft_consensus.cc:729] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.376821 12238 consensus_queue.cc:260] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.376893 12238 raft_consensus.cc:3037] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:33.383610 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6ec88552c643ed8da84c69f522dc56 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.383836 12325 tablet_replica.cc:330] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.383976 12325 raft_consensus.cc:2229] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:33.384119 12325 raft_consensus.cc:2258] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.384805 12325 ts_tablet_manager.cc:1905] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.386693 12325 ts_tablet_manager.cc:1918] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.386880 12325 log.cc:1196] T fb6ec88552c643ed8da84c69f522dc56 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/fb6ec88552c643ed8da84c69f522dc56
I20260921 18:46:33.392606 12238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "023aeebf56ac4343ae85ee0cd0143142"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.392715 12238 raft_consensus.cc:580] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Received request to step down
I20260921 18:46:33.392786 12238 raft_consensus.cc:3032] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:33.392823 12238 raft_consensus.cc:729] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.392932 12238 consensus_queue.cc:260] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.393009 12238 raft_consensus.cc:3037] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:33.398422 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 023aeebf56ac4343ae85ee0cd0143142 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.398590 12325 tablet_replica.cc:330] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.398725 12325 raft_consensus.cc:2229] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:33.398859 12325 raft_consensus.cc:2258] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.399474 12325 ts_tablet_manager.cc:1905] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.401203 12325 ts_tablet_manager.cc:1918] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.401288 12325 log.cc:1196] T 023aeebf56ac4343ae85ee0cd0143142 P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/023aeebf56ac4343ae85ee0cd0143142
I20260921 18:46:33.406975 12238 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "300ec264dcaa4e629d3e5b58cb9c075c"
dest_uuid: "76f21d7f54eb423ebddce8f5fdac8e41"
from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.407096 12238 raft_consensus.cc:580] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Received request to step down
I20260921 18:46:33.407167 12238 raft_consensus.cc:3032] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:33.407203 12238 raft_consensus.cc:729] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 76f21d7f54eb423ebddce8f5fdac8e41, State: Running, Role: LEADER
I20260921 18:46:33.407315 12238 consensus_queue.cc:260] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f21d7f54eb423ebddce8f5fdac8e41" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 33933 } }
I20260921 18:46:33.407392 12238 raft_consensus.cc:3037] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:33.412768 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 300ec264dcaa4e629d3e5b58cb9c075c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:47176
I20260921 18:46:33.412969 12325 tablet_replica.cc:330] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: stopping tablet replica
I20260921 18:46:33.413107 12325 raft_consensus.cc:2229] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:33.413231 12325 raft_consensus.cc:2258] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:33.413961 12325 ts_tablet_manager.cc:1905] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:33.415696 12325 ts_tablet_manager.cc:1918] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:33.415782 12325 log.cc:1196] T 300ec264dcaa4e629d3e5b58cb9c075c P 76f21d7f54eb423ebddce8f5fdac8e41: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/wals/300ec264dcaa4e629d3e5b58cb9c075c
I20260921 18:46:33.421608 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 12156
I20260921 18:46:33.427490 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 12065
2026-09-21T18:46:33Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (806 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20260921 18:46:33.447182 3218 test_util.cc:268] Using random seed: -1421474038
I20260921 18:46:33.447875 3218 ts_itest-base.cc:112] Starting cluster with:
I20260921 18:46:33.447919 3218 ts_itest-base.cc:113] --------------
I20260921 18:46:33.447932 3218 ts_itest-base.cc:114] 3 tablet servers
I20260921 18:46:33.447942 3218 ts_itest-base.cc:115] 3 replicas per TS
I20260921 18:46:33.447963 3218 ts_itest-base.cc:116] --------------
2026-09-21T18:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-21T18:46:33Z Disabled control of system clock
I20260921 18:46:33.456298 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:37509
--webserver_interface=127.3.36.190
--webserver_port=0
--builtin_ntp_servers=127.3.36.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.36.190:37509
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20260921 18:46:33.562781 12333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:33.563051 12333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:33.563113 12333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:33.567250 12333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260921 18:46:33.567332 12333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:33.567368 12333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260921 18:46:33.567417 12333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260921 18:46:33.572350 12333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.36.190:37509
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.36.190:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.36.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:33.573412 12333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:33.574797 12333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:33.582820 12338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:33.582820 12339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:33.582850 12341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:33.583055 12333 server_base.cc:1034] running on GCE node
I20260921 18:46:33.583786 12333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:33.584847 12333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:33.586030 12333 hybrid_clock.cc:648] HybridClock initialized: now 1790016393585971 us; error 46 us; skew 500 ppm
I20260921 18:46:33.587733 12333 webserver.cc:460] Webserver started at http://127.3.36.190:43823/ using document root <none> and password file <none>
I20260921 18:46:33.588292 12333 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:33.588353 12333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:33.588598 12333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:33.590500 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4518c74c051544febf230060229f0a29"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.591109 12333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4518c74c051544febf230060229f0a29"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.595016 12333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260921 18:46:33.597200 12347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.598387 12333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:33.598536 12333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4518c74c051544febf230060229f0a29"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.598641 12333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:33.609885 12333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:33.610487 12333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:33.610673 12333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:33.617214 12333 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.190:37509
I20260921 18:46:33.617220 12399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.190:37509 every 8 connection(s)
I20260921 18:46:33.618125 12333 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260921 18:46:33.621073 12400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:33.621207 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 12333
I20260921 18:46:33.621351 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260921 18:46:33.626338 12400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29: Bootstrap starting.
I20260921 18:46:33.628906 12400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:33.629760 12400 log.cc:824] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:33.631502 12400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29: No bootstrap required, opened a new log
I20260921 18:46:33.634946 12400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c74c051544febf230060229f0a29" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37509 } }
I20260921 18:46:33.635183 12400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260921 18:46:33.635264 12400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4518c74c051544febf230060229f0a29, State: Initialized, Role: FOLLOWER
I20260921 18:46:33.635653 12400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c74c051544febf230060229f0a29" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37509 } }
I20260921 18:46:33.635825 12400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260921 18:46:33.635907 12400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260921 18:46:33.636018 12400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:33.636806 12400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c74c051544febf230060229f0a29" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37509 } }
I20260921 18:46:33.637164 12400 leader_election.cc:302] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4518c74c051544febf230060229f0a29; no voters:
I20260921 18:46:33.637522 12400 leader_election.cc:288] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260921 18:46:33.637598 12405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:33.637779 12405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [term 1 LEADER]: Becoming Leader. State: Replica: 4518c74c051544febf230060229f0a29, State: Running, Role: LEADER
I20260921 18:46:33.638044 12405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c74c051544febf230060229f0a29" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37509 } }
I20260921 18:46:33.638723 12400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260921 18:46:33.639596 12407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4518c74c051544febf230060229f0a29. Latest consensus state: current_term: 1 leader_uuid: "4518c74c051544febf230060229f0a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c74c051544febf230060229f0a29" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37509 } } }
I20260921 18:46:33.639726 12407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:33.640012 12406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4518c74c051544febf230060229f0a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4518c74c051544febf230060229f0a29" member_type: VOTER last_known_addr { host: "127.3.36.190" port: 37509 } } }
I20260921 18:46:33.640097 12406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29 [sys.catalog]: This master's current role is: LEADER
I20260921 18:46:33.640218 12414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260921 18:46:33.643142 12414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260921 18:46:33.648075 12414 catalog_manager.cc:1347] Generated new cluster ID: 558d3c26ec144d30922b95ba54cd6e0f
I20260921 18:46:33.648142 12414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260921 18:46:33.674640 12414 catalog_manager.cc:1370] Generated new certificate authority record
I20260921 18:46:33.675608 12414 catalog_manager.cc:1504] Loading token signing keys...
I20260921 18:46:33.679806 12414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4518c74c051544febf230060229f0a29: Generated new TSK 0
I20260921 18:46:33.680438 12414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260921 18:46:33.687152 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.129:0
--local_ip_for_outbound_sockets=127.3.36.129
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37509
--builtin_ntp_servers=127.3.36.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20260921 18:46:33.803200 12424 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:33.803452 12424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:33.803472 12424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:33.803504 12424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:33.807533 12424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:33.807622 12424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.129
I20260921 18:46:33.812233 12424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.36.129
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.12424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.129
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:33.813191 12424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:33.814579 12424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:33.823436 12429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:33.823509 12430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:33.823519 12432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:33.824908 12424 server_base.cc:1034] running on GCE node
I20260921 18:46:33.825289 12424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:33.825927 12424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:33.827081 12424 hybrid_clock.cc:648] HybridClock initialized: now 1790016393827065 us; error 28 us; skew 500 ppm
I20260921 18:46:33.828792 12424 webserver.cc:460] Webserver started at http://127.3.36.129:35497/ using document root <none> and password file <none>
I20260921 18:46:33.829342 12424 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:33.829401 12424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:33.829664 12424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:33.831475 12424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "320be9abc4e64053ac798fc1a7b88ccf"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.832015 12424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "320be9abc4e64053ac798fc1a7b88ccf"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.835561 12424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:33.837754 12438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.838881 12424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260921 18:46:33.838995 12424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "320be9abc4e64053ac798fc1a7b88ccf"
format_stamp: "Formatted at 2026-09-21 18:46:33 on dist-test-slave-l4st"
I20260921 18:46:33.839128 12424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:33.855195 12424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:33.855796 12424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:33.856001 12424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:33.856539 12424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:33.857425 12424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:33.857581 12424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.857662 12424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:33.857699 12424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:33.866478 12424 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.129:41483
I20260921 18:46:33.866506 12551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.129:41483 every 8 connection(s)
I20260921 18:46:33.867321 12424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260921 18:46:33.873706 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 12424
I20260921 18:46:33.873799 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260921 18:46:33.876183 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.130:0
--local_ip_for_outbound_sockets=127.3.36.130
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37509
--builtin_ntp_servers=127.3.36.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:33.877588 12552 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37509
I20260921 18:46:33.877807 12552 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:33.878297 12552 heartbeater.cc:511] Master 127.3.36.190:37509 requested a full tablet report, sending...
I20260921 18:46:33.879721 12364 ts_manager.cc:194] Registered new tserver with Master: 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483)
I20260921 18:46:33.880875 12364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.129:49257
W20260921 18:46:33.988065 12555 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:33.988366 12555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:33.988395 12555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:33.988467 12555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:33.992581 12555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:33.992666 12555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.130
I20260921 18:46:33.997296 12555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.36.130
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.12555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.130
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:33.998303 12555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:33.999601 12555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:34.008785 12560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:34.008796 12563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:34.008791 12561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:34.009016 12555 server_base.cc:1034] running on GCE node
I20260921 18:46:34.009457 12555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:34.010123 12555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:34.011308 12555 hybrid_clock.cc:648] HybridClock initialized: now 1790016394011297 us; error 30 us; skew 500 ppm
I20260921 18:46:34.013016 12555 webserver.cc:460] Webserver started at http://127.3.36.130:42493/ using document root <none> and password file <none>
I20260921 18:46:34.013628 12555 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:34.013718 12555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:34.013916 12555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:34.015723 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "26392ab43382466e8013d85c56a421cd"
format_stamp: "Formatted at 2026-09-21 18:46:34 on dist-test-slave-l4st"
I20260921 18:46:34.016273 12555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "26392ab43382466e8013d85c56a421cd"
format_stamp: "Formatted at 2026-09-21 18:46:34 on dist-test-slave-l4st"
I20260921 18:46:34.019707 12555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260921 18:46:34.021999 12569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:34.023101 12555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260921 18:46:34.023226 12555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "26392ab43382466e8013d85c56a421cd"
format_stamp: "Formatted at 2026-09-21 18:46:34 on dist-test-slave-l4st"
I20260921 18:46:34.023324 12555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:34.040830 12555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:34.041460 12555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:34.041678 12555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:34.042275 12555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:34.043260 12555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:34.043329 12555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:34.043411 12555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:34.043459 12555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:34.052006 12555 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.130:45001
I20260921 18:46:34.052030 12682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.130:45001 every 8 connection(s)
I20260921 18:46:34.052866 12555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260921 18:46:34.062598 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 12555
I20260921 18:46:34.062748 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260921 18:46:34.063074 12683 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37509
I20260921 18:46:34.063337 12683 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:34.063899 12683 heartbeater.cc:511] Master 127.3.36.190:37509 requested a full tablet report, sending...
I20260921 18:46:34.064848 3218 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
/tmp/dist-test-tasko8_gC4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.36.131:0
--local_ip_for_outbound_sockets=127.3.36.131
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37509
--builtin_ntp_servers=127.3.36.148:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20260921 18:46:34.064890 12364 ts_manager.cc:194] Registered new tserver with Master: 26392ab43382466e8013d85c56a421cd (127.3.36.130:45001)
I20260921 18:46:34.065441 12364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.130:46365
W20260921 18:46:34.171944 12686 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260921 18:46:34.172183 12686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260921 18:46:34.172204 12686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260921 18:46:34.172232 12686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260921 18:46:34.176136 12686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260921 18:46:34.176216 12686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.36.131
I20260921 18:46:34.180893 12686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.36.148:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.36.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.36.131
--webserver_port=0
--tserver_master_addrs=127.3.36.190:37509
--never_fsync=true
--heap_profile_path=/tmp/kudu.12686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.36.131
--log_dir=/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 21 Sep 2026 18:08:17 UTC on ccycloud.quasar-olhkiv.root.comops.site
build id 4662545
I20260921 18:46:34.181854 12686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260921 18:46:34.183081 12686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260921 18:46:34.192165 12694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:34.192157 12692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260921 18:46:34.192162 12691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260921 18:46:34.192253 12686 server_base.cc:1034] running on GCE node
I20260921 18:46:34.192724 12686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260921 18:46:34.193324 12686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260921 18:46:34.194502 12686 hybrid_clock.cc:648] HybridClock initialized: now 1790016394194483 us; error 33 us; skew 500 ppm
I20260921 18:46:34.196208 12686 webserver.cc:460] Webserver started at http://127.3.36.131:33947/ using document root <none> and password file <none>
I20260921 18:46:34.196748 12686 fs_manager.cc:359] Metadata directory not provided
I20260921 18:46:34.196836 12686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260921 18:46:34.197046 12686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260921 18:46:34.198858 12686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4f7d1011b4cf471ca489839553021c67"
format_stamp: "Formatted at 2026-09-21 18:46:34 on dist-test-slave-l4st"
I20260921 18:46:34.199386 12686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4f7d1011b4cf471ca489839553021c67"
format_stamp: "Formatted at 2026-09-21 18:46:34 on dist-test-slave-l4st"
I20260921 18:46:34.202891 12686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260921 18:46:34.205061 12700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:34.206211 12686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:34.206298 12686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4f7d1011b4cf471ca489839553021c67"
format_stamp: "Formatted at 2026-09-21 18:46:34 on dist-test-slave-l4st"
I20260921 18:46:34.206382 12686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260921 18:46:34.227273 12686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260921 18:46:34.227829 12686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260921 18:46:34.227983 12686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260921 18:46:34.228485 12686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260921 18:46:34.229338 12686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260921 18:46:34.229382 12686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:34.229419 12686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260921 18:46:34.229434 12686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260921 18:46:34.237960 12686 rpc_server.cc:307] RPC server started. Bound to: 127.3.36.131:34985
I20260921 18:46:34.237982 12813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.36.131:34985 every 8 connection(s)
I20260921 18:46:34.238791 12686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260921 18:46:34.240525 3218 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu as pid 12686
I20260921 18:46:34.240645 3218 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260921 18:46:34.248761 12814 heartbeater.cc:347] Connected to a master server at 127.3.36.190:37509
I20260921 18:46:34.248976 12814 heartbeater.cc:464] Registering TS with master...
I20260921 18:46:34.249435 12814 heartbeater.cc:511] Master 127.3.36.190:37509 requested a full tablet report, sending...
I20260921 18:46:34.250394 12364 ts_manager.cc:194] Registered new tserver with Master: 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.250888 12364 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.36.131:54097
I20260921 18:46:34.254453 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:34.263514 3218 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260921 18:46:34.263563 3218 test_util.cc:268] Using random seed: -1420657652
I20260921 18:46:34.273582 12364 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39540:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260921 18:46:34.274994 12364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260921 18:46:34.296303 12608 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34dfb29cdf464b8c7ae1c457254c01 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260921 18:46:34.295568 12615 tablet_service.cc:1467] Processing CreateTablet for tablet ef795b1167b940c19531eb7b6e248882 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260921 18:46:34.296571 12609 tablet_service.cc:1467] Processing CreateTablet for tablet 39c23507908c4480a8df817e0baa6c1b (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260921 18:46:34.295686 12616 tablet_service.cc:1467] Processing CreateTablet for tablet 587489f2a0494ee49437de9e0d922217 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260921 18:46:34.296842 12610 tablet_service.cc:1467] Processing CreateTablet for tablet 5000e3b88ea14f2d977417df5950c9db (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260921 18:46:34.297082 12608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34dfb29cdf464b8c7ae1c457254c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.297029 12477 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34dfb29cdf464b8c7ae1c457254c01 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260921 18:46:34.296445 12486 tablet_service.cc:1467] Processing CreateTablet for tablet ef71a821eb154dfba5361495259f0f43 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260921 18:46:34.297144 12612 tablet_service.cc:1467] Processing CreateTablet for tablet 397e409f053e425fb3dbeb754182d8c7 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260921 18:46:34.297394 12613 tablet_service.cc:1467] Processing CreateTablet for tablet 84a44295fcda4948978dcced7e3608d1 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260921 18:46:34.297719 12616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587489f2a0494ee49437de9e0d922217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.297709 12614 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4fb53caffc45faaa728463b9ecca52 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260921 18:46:34.297782 12477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34dfb29cdf464b8c7ae1c457254c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.297938 12614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4fb53caffc45faaa728463b9ecca52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.298213 12486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef71a821eb154dfba5361495259f0f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.298681 12617 tablet_service.cc:1467] Processing CreateTablet for tablet ef71a821eb154dfba5361495259f0f43 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260921 18:46:34.298784 12615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef795b1167b940c19531eb7b6e248882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.298897 12617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef71a821eb154dfba5361495259f0f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.295570 12611 tablet_service.cc:1467] Processing CreateTablet for tablet 16c80c86d27c423eba504c2404ee9653 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260921 18:46:34.299057 12610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5000e3b88ea14f2d977417df5950c9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.299418 12611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c80c86d27c423eba504c2404ee9653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.299548 12613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a44295fcda4948978dcced7e3608d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.301681 12478 tablet_service.cc:1467] Processing CreateTablet for tablet 39c23507908c4480a8df817e0baa6c1b (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260921 18:46:34.302090 12479 tablet_service.cc:1467] Processing CreateTablet for tablet 5000e3b88ea14f2d977417df5950c9db (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260921 18:46:34.302474 12480 tablet_service.cc:1467] Processing CreateTablet for tablet 16c80c86d27c423eba504c2404ee9653 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260921 18:46:34.302830 12481 tablet_service.cc:1467] Processing CreateTablet for tablet 397e409f053e425fb3dbeb754182d8c7 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260921 18:46:34.303210 12482 tablet_service.cc:1467] Processing CreateTablet for tablet 84a44295fcda4948978dcced7e3608d1 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260921 18:46:34.303467 12612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397e409f053e425fb3dbeb754182d8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.303597 12484 tablet_service.cc:1467] Processing CreateTablet for tablet ef795b1167b940c19531eb7b6e248882 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260921 18:46:34.304013 12483 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4fb53caffc45faaa728463b9ecca52 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260921 18:46:34.304518 12485 tablet_service.cc:1467] Processing CreateTablet for tablet 587489f2a0494ee49437de9e0d922217 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260921 18:46:34.305639 12609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c23507908c4480a8df817e0baa6c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.307327 12739 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34dfb29cdf464b8c7ae1c457254c01 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260921 18:46:34.308281 12745 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4fb53caffc45faaa728463b9ecca52 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260921 18:46:34.308549 12740 tablet_service.cc:1467] Processing CreateTablet for tablet 39c23507908c4480a8df817e0baa6c1b (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260921 18:46:34.308683 12739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34dfb29cdf464b8c7ae1c457254c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.308981 12745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4fb53caffc45faaa728463b9ecca52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.308941 12741 tablet_service.cc:1467] Processing CreateTablet for tablet 5000e3b88ea14f2d977417df5950c9db (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260921 18:46:34.309168 12741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5000e3b88ea14f2d977417df5950c9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.307541 12746 tablet_service.cc:1467] Processing CreateTablet for tablet ef795b1167b940c19531eb7b6e248882 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260921 18:46:34.309859 12746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef795b1167b940c19531eb7b6e248882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.307334 12747 tablet_service.cc:1467] Processing CreateTablet for tablet 587489f2a0494ee49437de9e0d922217 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260921 18:46:34.310470 12747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587489f2a0494ee49437de9e0d922217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.310871 12742 tablet_service.cc:1467] Processing CreateTablet for tablet 16c80c86d27c423eba504c2404ee9653 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260921 18:46:34.311101 12742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c80c86d27c423eba504c2404ee9653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.312657 12743 tablet_service.cc:1467] Processing CreateTablet for tablet 397e409f053e425fb3dbeb754182d8c7 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260921 18:46:34.312879 12743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397e409f053e425fb3dbeb754182d8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.313179 12478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c23507908c4480a8df817e0baa6c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.313618 12482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a44295fcda4948978dcced7e3608d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.315552 12836 tablet_bootstrap.cc:492] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.315850 12483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4fb53caffc45faaa728463b9ecca52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.316179 12481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 397e409f053e425fb3dbeb754182d8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.316308 12744 tablet_service.cc:1467] Processing CreateTablet for tablet 84a44295fcda4948978dcced7e3608d1 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260921 18:46:34.316527 12744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a44295fcda4948978dcced7e3608d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.317870 12480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c80c86d27c423eba504c2404ee9653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.317875 12836 tablet_bootstrap.cc:654] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.318805 12748 tablet_service.cc:1467] Processing CreateTablet for tablet ef71a821eb154dfba5361495259f0f43 (DEFAULT_TABLE table=test-workload [id=bf793ca20555416a9ebe90b4fa9de2f0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260921 18:46:34.319016 12748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef71a821eb154dfba5361495259f0f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.319051 12836 log.cc:824] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:34.319734 12838 tablet_bootstrap.cc:492] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.319877 12484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef795b1167b940c19531eb7b6e248882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.320926 12740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c23507908c4480a8df817e0baa6c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.321877 12479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5000e3b88ea14f2d977417df5950c9db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.321882 12838 tablet_bootstrap.cc:654] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.322669 12838 log.cc:824] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:34.323797 12485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587489f2a0494ee49437de9e0d922217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260921 18:46:34.324859 12838 tablet_bootstrap.cc:492] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.325047 12838 ts_tablet_manager.cc:1397] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260921 18:46:34.325313 12836 tablet_bootstrap.cc:492] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.325552 12836 ts_tablet_manager.cc:1397] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260921 18:46:34.328928 12836 raft_consensus.cc:348] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.329192 12836 raft_consensus.cc:729] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.329088 12838 raft_consensus.cc:348] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.329339 12838 raft_consensus.cc:729] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.329550 12836 consensus_queue.cc:260] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.329648 12838 consensus_queue.cc:260] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.330302 12814 heartbeater.cc:503] Master 127.3.36.190:37509 was elected leader, sending a full tablet report...
I20260921 18:46:34.330616 12838 ts_tablet_manager.cc:1428] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260921 18:46:34.330770 12836 ts_tablet_manager.cc:1428] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260921 18:46:34.330938 12838 tablet_bootstrap.cc:492] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.331072 12683 heartbeater.cc:503] Master 127.3.36.190:37509 was elected leader, sending a full tablet report...
I20260921 18:46:34.331082 12836 tablet_bootstrap.cc:492] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.331881 12838 tablet_bootstrap.cc:654] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.332079 12836 tablet_bootstrap.cc:654] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.332933 12838 tablet_bootstrap.cc:492] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.333005 12838 ts_tablet_manager.cc:1397] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:34.333434 12837 tablet_bootstrap.cc:492] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.333583 12838 raft_consensus.cc:348] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.333681 12838 raft_consensus.cc:729] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.333771 12838 consensus_queue.cc:260] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.334111 12838 ts_tablet_manager.cc:1428] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.334264 12838 tablet_bootstrap.cc:492] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.334877 12836 tablet_bootstrap.cc:492] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.334954 12836 ts_tablet_manager.cc:1397] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260921 18:46:34.335186 12838 tablet_bootstrap.cc:654] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.335510 12836 raft_consensus.cc:348] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.335606 12836 raft_consensus.cc:729] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.335695 12836 consensus_queue.cc:260] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.336024 12836 ts_tablet_manager.cc:1428] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.336156 12838 tablet_bootstrap.cc:492] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.336198 12836 tablet_bootstrap.cc:492] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.336223 12838 ts_tablet_manager.cc:1397] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:34.336472 12837 tablet_bootstrap.cc:654] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.336896 12838 raft_consensus.cc:348] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.336993 12838 raft_consensus.cc:729] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.337073 12838 consensus_queue.cc:260] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.337280 12836 tablet_bootstrap.cc:654] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.337369 12838 ts_tablet_manager.cc:1428] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.337589 12838 tablet_bootstrap.cc:492] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.337854 12837 log.cc:824] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf: Log is configured to *not* fsync() on all Append() calls
I20260921 18:46:34.338466 12838 tablet_bootstrap.cc:654] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.339599 12838 tablet_bootstrap.cc:492] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.339701 12838 ts_tablet_manager.cc:1397] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:34.339984 12837 tablet_bootstrap.cc:492] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.340188 12837 ts_tablet_manager.cc:1397] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260921 18:46:34.340166 12838 raft_consensus.cc:348] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.340269 12838 raft_consensus.cc:729] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.340368 12838 consensus_queue.cc:260] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.340649 12838 ts_tablet_manager.cc:1428] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.340811 12838 tablet_bootstrap.cc:492] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.341643 12838 tablet_bootstrap.cc:654] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.342774 12838 tablet_bootstrap.cc:492] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.342844 12838 ts_tablet_manager.cc:1397] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260921 18:46:34.343214 12838 raft_consensus.cc:348] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.343313 12838 raft_consensus.cc:729] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.343495 12838 consensus_queue.cc:260] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.343854 12838 ts_tablet_manager.cc:1428] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.344027 12838 tablet_bootstrap.cc:492] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.344831 12838 tablet_bootstrap.cc:654] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.344733 12837 raft_consensus.cc:348] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.345026 12837 raft_consensus.cc:729] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.345371 12837 consensus_queue.cc:260] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.346067 12552 heartbeater.cc:503] Master 127.3.36.190:37509 was elected leader, sending a full tablet report...
I20260921 18:46:34.346453 12837 ts_tablet_manager.cc:1428] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260921 18:46:34.346688 12837 tablet_bootstrap.cc:492] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.347420 12838 tablet_bootstrap.cc:492] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.347492 12838 ts_tablet_manager.cc:1397] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260921 18:46:34.347636 12837 tablet_bootstrap.cc:654] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.348402 12838 raft_consensus.cc:348] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.348510 12838 raft_consensus.cc:729] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.348685 12836 tablet_bootstrap.cc:492] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.348603 12838 consensus_queue.cc:260] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.348774 12836 ts_tablet_manager.cc:1397] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20260921 18:46:34.348963 12838 ts_tablet_manager.cc:1428] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260921 18:46:34.348994 12837 tablet_bootstrap.cc:492] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.349113 12838 tablet_bootstrap.cc:492] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.349205 12837 ts_tablet_manager.cc:1397] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:34.349227 12836 raft_consensus.cc:348] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.349346 12836 raft_consensus.cc:729] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.349447 12836 consensus_queue.cc:260] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.349754 12836 ts_tablet_manager.cc:1428] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.349931 12836 tablet_bootstrap.cc:492] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.349997 12838 tablet_bootstrap.cc:654] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.350193 12837 raft_consensus.cc:348] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.350344 12837 raft_consensus.cc:729] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.350486 12837 consensus_queue.cc:260] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.350867 12836 tablet_bootstrap.cc:654] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.350988 12837 ts_tablet_manager.cc:1428] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:34.351125 12838 tablet_bootstrap.cc:492] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.351212 12838 ts_tablet_manager.cc:1397] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:34.351323 12837 tablet_bootstrap.cc:492] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.351696 12838 raft_consensus.cc:348] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.351809 12838 raft_consensus.cc:729] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.351910 12838 consensus_queue.cc:260] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.352002 12836 tablet_bootstrap.cc:492] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.352085 12836 ts_tablet_manager.cc:1397] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:34.352238 12838 ts_tablet_manager.cc:1428] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.352380 12838 tablet_bootstrap.cc:492] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.352547 12837 tablet_bootstrap.cc:654] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.352509 12836 raft_consensus.cc:348] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.352591 12836 raft_consensus.cc:729] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.352679 12836 consensus_queue.cc:260] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.352950 12836 ts_tablet_manager.cc:1428] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.353097 12836 tablet_bootstrap.cc:492] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.353315 12838 tablet_bootstrap.cc:654] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.354030 12836 tablet_bootstrap.cc:654] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.354367 12838 tablet_bootstrap.cc:492] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.354449 12838 ts_tablet_manager.cc:1397] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:34.354904 12838 raft_consensus.cc:348] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.354998 12838 raft_consensus.cc:729] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.355082 12838 consensus_queue.cc:260] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.355199 12836 tablet_bootstrap.cc:492] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.355232 12837 tablet_bootstrap.cc:492] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.355275 12836 ts_tablet_manager.cc:1397] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260921 18:46:34.355300 12837 ts_tablet_manager.cc:1397] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260921 18:46:34.355340 12838 ts_tablet_manager.cc:1428] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.355521 12838 tablet_bootstrap.cc:492] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.355705 12836 raft_consensus.cc:348] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.355731 12837 raft_consensus.cc:348] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.355818 12836 raft_consensus.cc:729] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.355826 12837 raft_consensus.cc:729] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.355926 12837 consensus_queue.cc:260] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.355919 12836 consensus_queue.cc:260] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.356199 12836 ts_tablet_manager.cc:1428] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.356236 12837 ts_tablet_manager.cc:1428] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.356334 12836 tablet_bootstrap.cc:492] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.356348 12838 tablet_bootstrap.cc:654] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.356406 12837 tablet_bootstrap.cc:492] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.357174 12836 tablet_bootstrap.cc:654] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.357280 12837 tablet_bootstrap.cc:654] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.358127 12838 tablet_bootstrap.cc:492] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.358271 12838 ts_tablet_manager.cc:1397] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260921 18:46:34.358346 12836 tablet_bootstrap.cc:492] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.358413 12836 ts_tablet_manager.cc:1397] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260921 18:46:34.358752 12838 raft_consensus.cc:348] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.358865 12838 raft_consensus.cc:729] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.358932 12836 raft_consensus.cc:348] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.358964 12838 consensus_queue.cc:260] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.359021 12836 raft_consensus.cc:729] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.359102 12836 consensus_queue.cc:260] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.359213 12838 ts_tablet_manager.cc:1428] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260921 18:46:34.359336 12836 ts_tablet_manager.cc:1428] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.359367 12838 tablet_bootstrap.cc:492] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67: Bootstrap starting.
I20260921 18:46:34.359459 12836 tablet_bootstrap.cc:492] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.360352 12836 tablet_bootstrap.cc:654] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.360620 12838 tablet_bootstrap.cc:654] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.361433 12837 tablet_bootstrap.cc:492] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.361557 12837 ts_tablet_manager.cc:1397] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260921 18:46:34.361897 12837 raft_consensus.cc:348] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.362000 12837 raft_consensus.cc:729] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.362107 12837 consensus_queue.cc:260] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.362368 12837 ts_tablet_manager.cc:1428] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.362524 12837 tablet_bootstrap.cc:492] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.363406 12837 tablet_bootstrap.cc:654] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.364259 12838 tablet_bootstrap.cc:492] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67: No bootstrap required, opened a new log
I20260921 18:46:34.364332 12838 ts_tablet_manager.cc:1397] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260921 18:46:34.364953 12836 tablet_bootstrap.cc:492] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.364933 12838 raft_consensus.cc:348] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.365032 12836 ts_tablet_manager.cc:1397] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260921 18:46:34.365025 12838 raft_consensus.cc:729] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d1011b4cf471ca489839553021c67, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.365113 12838 consensus_queue.cc:260] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.365399 12838 ts_tablet_manager.cc:1428] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.365425 12836 raft_consensus.cc:348] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.365546 12836 raft_consensus.cc:729] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.365636 12836 consensus_queue.cc:260] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.365868 12836 ts_tablet_manager.cc:1428] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.366000 12836 tablet_bootstrap.cc:492] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.366885 12836 tablet_bootstrap.cc:654] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.367913 12836 tablet_bootstrap.cc:492] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.368011 12836 ts_tablet_manager.cc:1397] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:34.368110 12837 tablet_bootstrap.cc:492] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.368191 12837 ts_tablet_manager.cc:1397] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260921 18:46:34.368459 12836 raft_consensus.cc:348] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.368553 12836 raft_consensus.cc:729] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.368574 12837 raft_consensus.cc:348] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.368664 12837 raft_consensus.cc:729] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.368646 12836 consensus_queue.cc:260] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.368726 12837 consensus_queue.cc:260] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.368825 12836 ts_tablet_manager.cc:1428] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.368983 12837 ts_tablet_manager.cc:1428] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.369038 12836 tablet_bootstrap.cc:492] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
W20260921 18:46:34.369023 12553 tablet.cc:2369] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:34.369124 12837 tablet_bootstrap.cc:492] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.370020 12837 tablet_bootstrap.cc:654] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.370149 12836 tablet_bootstrap.cc:654] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.371089 12837 tablet_bootstrap.cc:492] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.371176 12837 ts_tablet_manager.cc:1397] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:34.371402 12836 tablet_bootstrap.cc:492] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.371520 12836 ts_tablet_manager.cc:1397] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:34.372016 12837 raft_consensus.cc:348] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.372043 12836 raft_consensus.cc:348] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.372128 12837 raft_consensus.cc:729] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.372169 12836 raft_consensus.cc:729] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.372215 12837 consensus_queue.cc:260] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.372263 12836 consensus_queue.cc:260] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.372476 12837 ts_tablet_manager.cc:1428] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.372498 12836 ts_tablet_manager.cc:1428] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.372701 12837 tablet_bootstrap.cc:492] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.372745 12836 tablet_bootstrap.cc:492] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Bootstrap starting.
I20260921 18:46:34.373533 12836 tablet_bootstrap.cc:654] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.373605 12837 tablet_bootstrap.cc:654] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.374639 12837 tablet_bootstrap.cc:492] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.374716 12837 ts_tablet_manager.cc:1397] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260921 18:46:34.375079 12837 raft_consensus.cc:348] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.375237 12837 raft_consensus.cc:729] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.375330 12837 consensus_queue.cc:260] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.375581 12837 ts_tablet_manager.cc:1428] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.375734 12837 tablet_bootstrap.cc:492] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.376503 12837 tablet_bootstrap.cc:654] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.377296 12836 tablet_bootstrap.cc:492] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: No bootstrap required, opened a new log
I20260921 18:46:34.377394 12836 ts_tablet_manager.cc:1397] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260921 18:46:34.377717 12837 tablet_bootstrap.cc:492] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.377799 12837 ts_tablet_manager.cc:1397] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260921 18:46:34.377832 12836 raft_consensus.cc:348] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.377941 12836 raft_consensus.cc:729] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.378028 12836 consensus_queue.cc:260] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.378211 12837 raft_consensus.cc:348] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.378288 12836 ts_tablet_manager.cc:1428] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.378314 12837 raft_consensus.cc:729] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.378414 12837 consensus_queue.cc:260] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.378660 12837 ts_tablet_manager.cc:1428] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260921 18:46:34.378823 12837 tablet_bootstrap.cc:492] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.379711 12837 tablet_bootstrap.cc:654] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.381289 12837 tablet_bootstrap.cc:492] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.381366 12837 ts_tablet_manager.cc:1397] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260921 18:46:34.381814 12837 raft_consensus.cc:348] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.381911 12837 raft_consensus.cc:729] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.382010 12837 consensus_queue.cc:260] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.382263 12837 ts_tablet_manager.cc:1428] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260921 18:46:34.382440 12837 tablet_bootstrap.cc:492] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf: Bootstrap starting.
I20260921 18:46:34.383610 12837 tablet_bootstrap.cc:654] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf: Neither blocks nor log segments found. Creating new log.
I20260921 18:46:34.385260 12837 tablet_bootstrap.cc:492] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf: No bootstrap required, opened a new log
I20260921 18:46:34.385366 12837 ts_tablet_manager.cc:1397] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260921 18:46:34.385716 12837 raft_consensus.cc:348] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.385818 12837 raft_consensus.cc:729] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320be9abc4e64053ac798fc1a7b88ccf, State: Initialized, Role: FOLLOWER
I20260921 18:46:34.385913 12837 consensus_queue.cc:260] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.386193 12837 ts_tablet_manager.cc:1428] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260921 18:46:34.410881 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b34dfb29cdf464b8c7ae1c457254c01"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.411059 12637 raft_consensus.cc:482] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.411149 12637 raft_consensus.cc:3037] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.412037 12637 raft_consensus.cc:504] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.412986 12637 leader_election.cc:288] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.417999 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34dfb29cdf464b8c7ae1c457254c01" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.418040 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34dfb29cdf464b8c7ae1c457254c01" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.418217 12768 raft_consensus.cc:3037] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.418231 12506 raft_consensus.cc:3037] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.419341 12506 raft_consensus.cc:2454] T 1b34dfb29cdf464b8c7ae1c457254c01 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.419346 12768 raft_consensus.cc:2454] T 1b34dfb29cdf464b8c7ae1c457254c01 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.419730 12571 leader_election.cc:302] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.419984 12847 raft_consensus.cc:2781] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.420104 12847 raft_consensus.cc:686] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.420408 12847 consensus_queue.cc:237] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.423655 12362 catalog_manager.cc:5495] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.429237 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84a44295fcda4948978dcced7e3608d1"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.429345 12637 raft_consensus.cc:482] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.429382 12637 raft_consensus.cc:3037] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.430173 12637 raft_consensus.cc:504] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.430459 12637 leader_election.cc:288] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.430754 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a44295fcda4948978dcced7e3608d1" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.430784 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a44295fcda4948978dcced7e3608d1" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.430900 12768 raft_consensus.cc:3037] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.430898 12506 raft_consensus.cc:3037] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.431775 12506 raft_consensus.cc:2454] T 84a44295fcda4948978dcced7e3608d1 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.431777 12768 raft_consensus.cc:2454] T 84a44295fcda4948978dcced7e3608d1 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.432045 12571 leader_election.cc:302] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.432238 12847 raft_consensus.cc:2781] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.432336 12847 raft_consensus.cc:686] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.432467 12847 consensus_queue.cc:237] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.434031 12362 catalog_manager.cc:5495] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.439306 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef71a821eb154dfba5361495259f0f43"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.439436 12637 raft_consensus.cc:482] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.439517 12637 raft_consensus.cc:3037] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.440277 12637 raft_consensus.cc:504] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.440583 12637 leader_election.cc:288] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.440914 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef71a821eb154dfba5361495259f0f43" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.441005 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef71a821eb154dfba5361495259f0f43" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.441051 12768 raft_consensus.cc:3037] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.441130 12506 raft_consensus.cc:3037] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.441960 12768 raft_consensus.cc:2454] T ef71a821eb154dfba5361495259f0f43 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.442201 12506 raft_consensus.cc:2454] T ef71a821eb154dfba5361495259f0f43 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.442247 12571 leader_election.cc:302] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.442404 12847 raft_consensus.cc:2781] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.442492 12847 raft_consensus.cc:686] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.442625 12847 consensus_queue.cc:237] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.444226 12362 catalog_manager.cc:5495] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.449331 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39c23507908c4480a8df817e0baa6c1b"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.449424 12637 raft_consensus.cc:482] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.449458 12637 raft_consensus.cc:3037] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.450217 12637 raft_consensus.cc:504] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.450496 12637 leader_election.cc:288] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.450803 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c23507908c4480a8df817e0baa6c1b" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.450819 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c23507908c4480a8df817e0baa6c1b" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.450924 12768 raft_consensus.cc:3037] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.450924 12506 raft_consensus.cc:3037] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.451783 12506 raft_consensus.cc:2454] T 39c23507908c4480a8df817e0baa6c1b P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.451783 12768 raft_consensus.cc:2454] T 39c23507908c4480a8df817e0baa6c1b P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.452064 12571 leader_election.cc:302] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.452236 12847 raft_consensus.cc:2781] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.452322 12847 raft_consensus.cc:686] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.452464 12847 consensus_queue.cc:237] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.454044 12362 catalog_manager.cc:5495] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.458949 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "587489f2a0494ee49437de9e0d922217"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.459072 12637 raft_consensus.cc:482] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.459148 12637 raft_consensus.cc:3037] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.459837 12637 raft_consensus.cc:504] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.460162 12637 leader_election.cc:288] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.460484 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587489f2a0494ee49437de9e0d922217" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.460508 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587489f2a0494ee49437de9e0d922217" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.460596 12506 raft_consensus.cc:3037] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.460610 12768 raft_consensus.cc:3037] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.461459 12768 raft_consensus.cc:2454] T 587489f2a0494ee49437de9e0d922217 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.461459 12506 raft_consensus.cc:2454] T 587489f2a0494ee49437de9e0d922217 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.461751 12571 leader_election.cc:302] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.461913 12847 raft_consensus.cc:2781] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.462000 12847 raft_consensus.cc:686] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.462121 12847 consensus_queue.cc:237] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.463665 12362 catalog_manager.cc:5495] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.468634 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b4fb53caffc45faaa728463b9ecca52"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.468722 12637 raft_consensus.cc:482] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.468758 12637 raft_consensus.cc:3037] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.469403 12637 raft_consensus.cc:504] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.469743 12637 leader_election.cc:288] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.470001 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4fb53caffc45faaa728463b9ecca52" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.470084 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4fb53caffc45faaa728463b9ecca52" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.470122 12506 raft_consensus.cc:3037] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.470211 12768 raft_consensus.cc:3037] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.470846 12506 raft_consensus.cc:2454] T 1b4fb53caffc45faaa728463b9ecca52 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.471086 12768 raft_consensus.cc:2454] T 1b4fb53caffc45faaa728463b9ecca52 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.471114 12570 leader_election.cc:302] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 320be9abc4e64053ac798fc1a7b88ccf; no voters:
I20260921 18:46:34.471287 12847 raft_consensus.cc:2781] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.471370 12847 raft_consensus.cc:686] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.471504 12847 consensus_queue.cc:237] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.472991 12362 catalog_manager.cc:5495] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.478003 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "16c80c86d27c423eba504c2404ee9653"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.478119 12637 raft_consensus.cc:482] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.478197 12637 raft_consensus.cc:3037] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.478878 12637 raft_consensus.cc:504] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.479183 12637 leader_election.cc:288] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.479434 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c80c86d27c423eba504c2404ee9653" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.479548 12506 raft_consensus.cc:3037] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.479513 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c80c86d27c423eba504c2404ee9653" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.479631 12768 raft_consensus.cc:3037] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.480391 12506 raft_consensus.cc:2454] T 16c80c86d27c423eba504c2404ee9653 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.480407 12768 raft_consensus.cc:2454] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.480676 12571 leader_election.cc:302] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.480851 12847 raft_consensus.cc:2781] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.480932 12847 raft_consensus.cc:686] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.481051 12847 consensus_queue.cc:237] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.482543 12362 catalog_manager.cc:5495] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.487398 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5000e3b88ea14f2d977417df5950c9db"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.487486 12637 raft_consensus.cc:482] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.487565 12637 raft_consensus.cc:3037] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.488255 12637 raft_consensus.cc:504] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.488579 12637 leader_election.cc:288] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.488849 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5000e3b88ea14f2d977417df5950c9db" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.488929 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5000e3b88ea14f2d977417df5950c9db" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.488957 12506 raft_consensus.cc:3037] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.489024 12768 raft_consensus.cc:3037] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
W20260921 18:46:34.489717 12815 tablet.cc:2369] T 16c80c86d27c423eba504c2404ee9653 P 4f7d1011b4cf471ca489839553021c67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:34.489818 12768 raft_consensus.cc:2454] T 5000e3b88ea14f2d977417df5950c9db P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.489818 12506 raft_consensus.cc:2454] T 5000e3b88ea14f2d977417df5950c9db P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.490113 12571 leader_election.cc:302] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 320be9abc4e64053ac798fc1a7b88ccf, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.490283 12847 raft_consensus.cc:2781] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.490372 12847 raft_consensus.cc:686] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.490480 12847 consensus_queue.cc:237] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.491997 12362 catalog_manager.cc:5495] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.496968 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "397e409f053e425fb3dbeb754182d8c7"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.497054 12637 raft_consensus.cc:482] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.497089 12637 raft_consensus.cc:3037] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.497781 12637 raft_consensus.cc:504] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.498059 12637 leader_election.cc:288] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.498314 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397e409f053e425fb3dbeb754182d8c7" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.498342 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "397e409f053e425fb3dbeb754182d8c7" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.498425 12506 raft_consensus.cc:3037] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.498433 12768 raft_consensus.cc:3037] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.499228 12506 raft_consensus.cc:2454] T 397e409f053e425fb3dbeb754182d8c7 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.499228 12768 raft_consensus.cc:2454] T 397e409f053e425fb3dbeb754182d8c7 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.499475 12571 leader_election.cc:302] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.499712 12841 raft_consensus.cc:2781] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.499841 12841 raft_consensus.cc:686] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.499962 12841 consensus_queue.cc:237] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.501432 12362 catalog_manager.cc:5495] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.506486 12637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef795b1167b940c19531eb7b6e248882"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.506574 12637 raft_consensus.cc:482] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260921 18:46:34.506624 12637 raft_consensus.cc:3037] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.507254 12637 raft_consensus.cc:504] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.507524 12637 leader_election.cc:288] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Requested vote from peers 320be9abc4e64053ac798fc1a7b88ccf (127.3.36.129:41483), 4f7d1011b4cf471ca489839553021c67 (127.3.36.131:34985)
I20260921 18:46:34.507789 12506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef795b1167b940c19531eb7b6e248882" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "320be9abc4e64053ac798fc1a7b88ccf"
I20260921 18:46:34.507860 12768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef795b1167b940c19531eb7b6e248882" candidate_uuid: "26392ab43382466e8013d85c56a421cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7d1011b4cf471ca489839553021c67"
I20260921 18:46:34.507905 12506 raft_consensus.cc:3037] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.507990 12768 raft_consensus.cc:3037] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67 [term 0 FOLLOWER]: Advancing to term 1
I20260921 18:46:34.508795 12768 raft_consensus.cc:2454] T ef795b1167b940c19531eb7b6e248882 P 4f7d1011b4cf471ca489839553021c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.508826 12506 raft_consensus.cc:2454] T ef795b1167b940c19531eb7b6e248882 P 320be9abc4e64053ac798fc1a7b88ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26392ab43382466e8013d85c56a421cd in term 1.
I20260921 18:46:34.509058 12571 leader_election.cc:302] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26392ab43382466e8013d85c56a421cd, 4f7d1011b4cf471ca489839553021c67; no voters:
I20260921 18:46:34.509258 12841 raft_consensus.cc:2781] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Leader election won for term 1
I20260921 18:46:34.509362 12841 raft_consensus.cc:686] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Leader. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.509496 12841 consensus_queue.cc:237] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.510892 12362 catalog_manager.cc:5495] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 26392ab43382466e8013d85c56a421cd (127.3.36.130). New cstate: current_term: 1 leader_uuid: "26392ab43382466e8013d85c56a421cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260921 18:46:34.516805 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b34dfb29cdf464b8c7ae1c457254c01 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.517221 12887 tablet_replica.cc:330] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.517593 12887 raft_consensus.cc:2229] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:34.517925 12887 pending_rounds.cc:62] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.518030 12887 pending_rounds.cc:68] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151548776448 op_type: NO_OP noop_request { }
I20260921 18:46:34.518159 12887 raft_consensus.cc:2866] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.518245 12887 raft_consensus.cc:2258] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.519310 12887 ts_tablet_manager.cc:1905] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.521178 12887 ts_tablet_manager.cc:1918] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.521279 12887 log.cc:1196] T 1b34dfb29cdf464b8c7ae1c457254c01 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b34dfb29cdf464b8c7ae1c457254c01
I20260921 18:46:34.527000 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a44295fcda4948978dcced7e3608d1 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.527168 12887 tablet_replica.cc:330] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.527261 12887 raft_consensus.cc:2229] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:34.527393 12887 pending_rounds.cc:62] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.527436 12887 pending_rounds.cc:68] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151596621824 op_type: NO_OP noop_request { }
I20260921 18:46:34.527480 12887 raft_consensus.cc:2866] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.527522 12887 raft_consensus.cc:2258] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.528025 12887 ts_tablet_manager.cc:1905] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.529422 12887 ts_tablet_manager.cc:1918] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.529563 12887 log.cc:1196] T 84a44295fcda4948978dcced7e3608d1 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/84a44295fcda4948978dcced7e3608d1
I20260921 18:46:34.535028 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet ef71a821eb154dfba5361495259f0f43 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.535187 12887 tablet_replica.cc:330] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.535275 12887 raft_consensus.cc:2229] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:34.535372 12887 pending_rounds.cc:62] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.535396 12887 pending_rounds.cc:68] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151638544384 op_type: NO_OP noop_request { }
I20260921 18:46:34.535435 12887 raft_consensus.cc:2866] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.535475 12887 raft_consensus.cc:2258] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.535949 12887 ts_tablet_manager.cc:1905] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.537290 12887 ts_tablet_manager.cc:1918] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.537349 12887 log.cc:1196] T ef71a821eb154dfba5361495259f0f43 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef71a821eb154dfba5361495259f0f43
I20260921 18:46:34.543210 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c23507908c4480a8df817e0baa6c1b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.543358 12887 tablet_replica.cc:330] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.543439 12887 raft_consensus.cc:2229] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:34.543534 12887 pending_rounds.cc:62] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.543556 12887 pending_rounds.cc:68] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151678566400 op_type: NO_OP noop_request { }
I20260921 18:46:34.543602 12887 raft_consensus.cc:2866] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.543640 12887 raft_consensus.cc:2258] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.544090 12887 ts_tablet_manager.cc:1905] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.545413 12887 ts_tablet_manager.cc:1918] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.545542 12887 log.cc:1196] T 39c23507908c4480a8df817e0baa6c1b P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/39c23507908c4480a8df817e0baa6c1b
I20260921 18:46:34.551111 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 587489f2a0494ee49437de9e0d922217 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.551271 12887 tablet_replica.cc:330] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.551347 12887 raft_consensus.cc:2229] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Raft consensus shutting down.
I20260921 18:46:34.551437 12887 pending_rounds.cc:62] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.551479 12887 pending_rounds.cc:68] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151718215680 op_type: NO_OP noop_request { }
I20260921 18:46:34.551515 12887 raft_consensus.cc:2866] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.551553 12887 raft_consensus.cc:2258] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.552037 12887 ts_tablet_manager.cc:1905] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.553329 12887 ts_tablet_manager.cc:1918] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.553387 12887 log.cc:1196] T 587489f2a0494ee49437de9e0d922217 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/587489f2a0494ee49437de9e0d922217
W20260921 18:46:34.553896 12684 tablet.cc:2369] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260921 18:46:34.559094 12637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b4fb53caffc45faaa728463b9ecca52"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.559214 12637 raft_consensus.cc:580] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Received request to step down
I20260921 18:46:34.559247 12637 raft_consensus.cc:3032] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:34.559273 12637 raft_consensus.cc:729] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.559368 12637 consensus_queue.cc:260] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.559461 12637 raft_consensus.cc:3037] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:34.564890 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4fb53caffc45faaa728463b9ecca52 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.565042 12887 tablet_replica.cc:330] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.565127 12887 raft_consensus.cc:2229] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:34.565192 12887 pending_rounds.cc:62] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.565212 12887 pending_rounds.cc:68] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151756337152 op_type: NO_OP noop_request { }
I20260921 18:46:34.565253 12887 raft_consensus.cc:2866] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.565289 12887 raft_consensus.cc:2258] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.565843 12887 ts_tablet_manager.cc:1905] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.567184 12887 ts_tablet_manager.cc:1918] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.567242 12887 log.cc:1196] T 1b4fb53caffc45faaa728463b9ecca52 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b4fb53caffc45faaa728463b9ecca52
I20260921 18:46:34.572557 12637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "16c80c86d27c423eba504c2404ee9653"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.572654 12637 raft_consensus.cc:580] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Received request to step down
I20260921 18:46:34.572685 12637 raft_consensus.cc:3032] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:34.572705 12637 raft_consensus.cc:729] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.572803 12637 consensus_queue.cc:260] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.572926 12637 raft_consensus.cc:3037] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:34.577908 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c80c86d27c423eba504c2404ee9653 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.578090 12887 tablet_replica.cc:330] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.578208 12887 raft_consensus.cc:2229] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:34.578295 12887 pending_rounds.cc:62] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.578328 12887 pending_rounds.cc:68] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151795589120 op_type: NO_OP noop_request { }
I20260921 18:46:34.578431 12887 raft_consensus.cc:2866] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.578487 12887 raft_consensus.cc:2258] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.579022 12887 ts_tablet_manager.cc:1905] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.580525 12887 ts_tablet_manager.cc:1918] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.580595 12887 log.cc:1196] T 16c80c86d27c423eba504c2404ee9653 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/16c80c86d27c423eba504c2404ee9653
I20260921 18:46:34.586026 12637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5000e3b88ea14f2d977417df5950c9db"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.586129 12637 raft_consensus.cc:580] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Received request to step down
I20260921 18:46:34.586198 12637 raft_consensus.cc:3032] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:34.586234 12637 raft_consensus.cc:729] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.586337 12637 consensus_queue.cc:260] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.586470 12637 raft_consensus.cc:3037] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:34.591738 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 5000e3b88ea14f2d977417df5950c9db with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.591918 12887 tablet_replica.cc:330] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.592041 12887 raft_consensus.cc:2229] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:34.592155 12887 pending_rounds.cc:62] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.592192 12887 pending_rounds.cc:68] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151834230784 op_type: NO_OP noop_request { }
I20260921 18:46:34.592248 12887 raft_consensus.cc:2866] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.592298 12887 raft_consensus.cc:2258] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.592870 12887 ts_tablet_manager.cc:1905] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.594491 12887 ts_tablet_manager.cc:1918] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.594551 12887 log.cc:1196] T 5000e3b88ea14f2d977417df5950c9db P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/5000e3b88ea14f2d977417df5950c9db
I20260921 18:46:34.600239 12637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "397e409f053e425fb3dbeb754182d8c7"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.600338 12637 raft_consensus.cc:580] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Received request to step down
I20260921 18:46:34.600401 12637 raft_consensus.cc:3032] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:34.600441 12637 raft_consensus.cc:729] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.600562 12637 consensus_queue.cc:260] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.600688 12637 raft_consensus.cc:3037] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:34.606132 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet 397e409f053e425fb3dbeb754182d8c7 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.606350 12887 tablet_replica.cc:330] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.606488 12887 raft_consensus.cc:2229] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:34.606585 12887 pending_rounds.cc:62] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.606624 12887 pending_rounds.cc:68] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151873163264 op_type: NO_OP noop_request { }
I20260921 18:46:34.606698 12887 raft_consensus.cc:2866] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.606753 12887 raft_consensus.cc:2258] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.607265 12887 ts_tablet_manager.cc:1905] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.608829 12887 ts_tablet_manager.cc:1918] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.608891 12887 log.cc:1196] T 397e409f053e425fb3dbeb754182d8c7 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/397e409f053e425fb3dbeb754182d8c7
I20260921 18:46:34.614569 12637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef795b1167b940c19531eb7b6e248882"
dest_uuid: "26392ab43382466e8013d85c56a421cd"
from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.614681 12637 raft_consensus.cc:580] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Received request to step down
I20260921 18:46:34.614711 12637 raft_consensus.cc:3032] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Stepping down as leader of term 1
I20260921 18:46:34.614737 12637 raft_consensus.cc:729] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26392ab43382466e8013d85c56a421cd, State: Running, Role: LEADER
I20260921 18:46:34.614827 12637 consensus_queue.cc:260] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320be9abc4e64053ac798fc1a7b88ccf" member_type: VOTER last_known_addr { host: "127.3.36.129" port: 41483 } } peers { permanent_uuid: "26392ab43382466e8013d85c56a421cd" member_type: VOTER last_known_addr { host: "127.3.36.130" port: 45001 } } peers { permanent_uuid: "4f7d1011b4cf471ca489839553021c67" member_type: VOTER last_known_addr { host: "127.3.36.131" port: 34985 } }
I20260921 18:46:34.614943 12637 raft_consensus.cc:3037] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 1 FOLLOWER]: Advancing to term 2
I20260921 18:46:34.620240 12608 tablet_service.cc:1514] Processing DeleteTablet for tablet ef795b1167b940c19531eb7b6e248882 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39894
I20260921 18:46:34.620393 12887 tablet_replica.cc:330] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: stopping tablet replica
I20260921 18:46:34.620502 12887 raft_consensus.cc:2229] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260921 18:46:34.620608 12887 pending_rounds.cc:62] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Trying to abort 1 pending ops.
I20260921 18:46:34.620641 12887 pending_rounds.cc:68] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7331907151912329216 op_type: NO_OP noop_request { }
I20260921 18:46:34.620702 12887 raft_consensus.cc:2866] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260921 18:46:34.620738 12887 raft_consensus.cc:2258] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260921 18:46:34.621269 12887 ts_tablet_manager.cc:1905] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260921 18:46:34.623268 12887 ts_tablet_manager.cc:1918] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260921 18:46:34.623337 12887 log.cc:1196] T ef795b1167b940c19531eb7b6e248882 P 26392ab43382466e8013d85c56a421cd: Deleting WAL directory at /tmp/dist-test-tasko8_gC4/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1790016227753869-3218-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef795b1167b940c19531eb7b6e248882
I20260921 18:46:34.632239 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 12424
I20260921 18:46:34.638343 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 12555
I20260921 18:46:34.643954 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 12686
I20260921 18:46:34.649668 3218 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko8_gC4/build/debug/bin/kudu with pid 12333
2026-09-21T18:46:34Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (1226 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (2033 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (166896 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20260921 18:46:34.674676 3218 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 6 messages since previous log ~10 seconds ago